Jun 09 00:51:00.356591 np0000002381 systemd[1]: Started devstack@neutron-ovn-maintenance-worker.service - Devstack devstack@neutron-ovn-maintenance-worker.service. Jun 09 00:51:01.962755 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.common.config [-] Logging enabled! Jun 09 00:51:01.963251 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker version 28.1.0.dev375 Jun 09 00:51:01.963251 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini {{(pid=88492) setup_logging /opt/stack/neutron/neutron/common/config.py:125}} Jun 09 00:51:01.963776 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.server.ovn_maintenance [-] OVN maintenance process starting... Jun 09 00:51:01.963899 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "plugin-directory" by "neutron_lib.plugins.directory._create_plugin_directory" {{(pid=88492) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} Jun 09 00:51:01.964025 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" acquired by "neutron_lib.plugins.directory._create_plugin_directory" :: waited 0.000s {{(pid=88492) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} Jun 09 00:51:01.964091 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" "released" by "neutron_lib.plugins.directory._create_plugin_directory" :: held 0.000s {{(pid=88492) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} Jun 09 00:51:01.964178 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "manager" by "neutron.manager.NeutronManager._create_instance" {{(pid=88492) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} Jun 09 00:51:01.964285 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG oslo_concurrency.lockutils [-] Lock "manager" acquired by "neutron.manager.NeutronManager._create_instance" :: waited 0.000s {{(pid=88492) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} Jun 09 00:51:01.964408 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.manager [-] Loading core plugin: ml2 Jun 09 00:51:02.353966 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.354089 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.354122 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.354700 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.354759 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.354822 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.394202 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.394310 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.394435 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.402852 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.402940 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.403027 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.451070 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.451146 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.451265 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.461370 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.461403 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.461477 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.531604 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.531689 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.531809 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment precommit_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.531902 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: network precommit_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.532212 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.596765 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.596866 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.596934 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.596996 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597061 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597126 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597198 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597289 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597358 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597433 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597494 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network precommit_delete 0, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597561 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port provisioning_complete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597624 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597683 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597747 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597808 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597861 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete 0, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597923 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.597976 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.598042 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.598106 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.598213 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.598281 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.598334 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.598644 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: network configured {{(pid=88492) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 09 00:51:02.598709 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: port configured {{(pid=88492) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 09 00:51:02.598762 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnet configured {{(pid=88492) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 09 00:51:02.598816 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnetpool configured {{(pid=88492) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 09 00:51:02.598902 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group configured {{(pid=88492) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 09 00:51:02.598999 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group_rule configured {{(pid=88492) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 09 00:51:02.599259 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'geneve'] Jun 09 00:51:02.600151 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['public'] Jun 09 00:51:02.600785 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete Jun 09 00:51:02.601402 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: OrderedDict({'public': [(1, 4094)]}) Jun 09 00:51:02.601510 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve', 'local', 'vlan'] Jun 09 00:51:02.601684 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve', 'local', 'vlan']) Jun 09 00:51:02.601814 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] Project network_types: ['geneve'] Jun 09 00:51:02.601814 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos'] Jun 09 00:51:02.602353 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos'] Jun 09 00:51:02.602403 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos'] Jun 09 00:51:02.602513 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn'] Jun 09 00:51:02.645214 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn'] Jun 09 00:51:02.645214 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn'] Jun 09 00:51:02.645214 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling. Jun 09 00:51:02.645214 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_response 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.645214 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_response 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:02.664852 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat' Jun 09 00:51:02.664852 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete Jun 09 00:51:02.664852 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'geneve' Jun 09 00:51:02.664938 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.drivers.type_tunnel [-] geneve ID ranges: [(1, 65536)] Jun 09 00:51:02.705080 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO dbcounter [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Registered counter for database neutron Jun 09 00:51:02.743008 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG oslo_db.sqlalchemy.engines [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=88492) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:325}} Jun 09 00:51:03.153643 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG dbcounter [-] [88492] Writer thread running {{(pid=88492) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} Jun 09 00:51:03.314747 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Initializing driver for type 'local' Jun 09 00:51:03.314848 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Initializing driver for type 'vlan' Jun 09 00:51:03.413834 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.drivers.type_vlan [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] VlanTypeDriver initialization complete Jun 09 00:51:03.413834 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Initializing extension driver 'port_security' Jun 09 00:51:03.413834 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.extensions.port_security [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] PortSecurityExtensionDriver initialization complete Jun 09 00:51:03.413834 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Initializing extension driver 'qos' Jun 09 00:51:03.414086 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.plugins.ml2.extensions.qos [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] QosExtensionDriver initialization complete {{(pid=88492) initialize /opt/stack/neutron/neutron/plugins/ml2/extensions/qos.py:31}} Jun 09 00:51:03.414086 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Initializing mechanism driver 'ovn' Jun 09 00:51:03.414086 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Starting OVNMechanismDriver Jun 09 00:51:03.415921 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > process before_spawn 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.415921 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > process after_init 55550000, True {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.415921 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > segment after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.415921 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > segment after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.416038 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > segment after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.416038 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > allowed_address_pair before_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.416280 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group precommit_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.416322 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.417445 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.417445 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group precommit_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.417445 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.417445 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group_rule precommit_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.417613 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.417613 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group_rule before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.417613 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > address_group after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.417613 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > address_group after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.417767 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > address_group after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.417767 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > qos_plugin after_init 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.417767 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > trunk_plugin after_init 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.417767 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > logging-plugin after_init 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.418649 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.services.logapi.drivers.ovn.driver [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] OVN logging driver registered Jun 09 00:51:03.418649 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: ._register_pvlan_driver at 0x7745626fae80> pvlan_plugin before_spawn 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.418649 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: process after_init 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.420121 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.plugins.ml2.plugin [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] neutron.plugins.ml2.plugin.Ml2Plugin method _start_rpc_notifiers called with arguments () {} {{(pid=88492) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Jun 09 00:51:03.420121 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > port after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.420228 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > port after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.420266 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > port after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.420507 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > subnet after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.420507 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > subnet after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.420507 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > subnet after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.420632 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > network after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.420632 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > network after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.420699 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > network after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.422472 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.422472 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.422472 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.422472 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.422623 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group_rule after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.422623 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.422623 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > address_group after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.422623 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > address_group after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.422771 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > address_group after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.422771 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.plugins.ml2.ovo_rpc [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ML2 OVO RPC backend initialized. {{(pid=88492) __init__ /opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py:117}} Jun 09 00:51:03.422771 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.quota [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded quota_driver: . Jun 09 00:51:03.422771 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.plugin [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Modular L2 Plugin initialization complete Jun 09 00:51:03.422771 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading services supported by the core plugin {{(pid=88492) _load_services_from_core_plugin /opt/stack/neutron/neutron/manager.py:170}} Jun 09 00:51:03.422771 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.plugins.ml2.managers [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Got port-security extension from driver 'port_security' Jun 09 00:51:03.422771 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading service plugins: ['ovn-router', 'auto_allocate', 'tag', 'timestamp', 'network_ip_availability', 'flavors', 'revisions'] {{(pid=88492) _load_service_plugins /opt/stack/neutron/neutron/manager.py:195}} Jun 09 00:51:03.422771 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading Plugin: ovn-router Jun 09 00:51:03.432873 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.433015 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: port before_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.433113 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: port after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.433229 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > port precommit_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.433345 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: port before_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.433459 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: port before_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.433558 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router precommit_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.433663 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router precommit_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.433774 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router precommit_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.433888 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: subnet after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.434096 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.quota.resource_registry [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Tracking information for resource: router configured {{(pid=88492) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 09 00:51:03.434186 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.quota.resource_registry [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Tracking information for resource: floatingip configured {{(pid=88492) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 09 00:51:03.434300 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.services.ovn_l3.plugin [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Starting OVNL3RouterPlugin Jun 09 00:51:03.434787 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > port_forwarding_plugin after_init 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.434787 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router before_create 55550100, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.434888 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router precommit_delete 55550100, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.434995 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router precommit_create 55550100, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.435112 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router precommit_update 55550100, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.438748 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.services.provider_configuration [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Service providers = [] {{(pid=88492) parse_service_provider_opt /opt/stack/neutron/neutron/services/provider_configuration.py:170}} Jun 09 00:51:03.438856 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.db.servicetype_db [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Adding provider configuration for service L3_ROUTER_NAT {{(pid=88492) add_provider_configuration /opt/stack/neutron/neutron/db/servicetype_db.py:45}} Jun 09 00:51:03.440466 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router precommit_create 55549900, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.440685 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router_interface after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.440754 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > floatingip after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.440822 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > floatingip precommit_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.440921 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > floatingip precommit_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.441003 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > floatingip after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.441078 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > floatingip after_status_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.441154 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > floatingip after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.441273 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > port_forwarding after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.441353 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router_interface after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.441444 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router after_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.441523 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router precommit_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.441603 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router after_delete 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.441673 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > router after_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.441761 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.services.service_base [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded 'neutron.services.ovn_l3.service_providers.ovn.OvnDriver' provider for service L3_ROUTER_NAT {{(pid=88492) load_drivers /opt/stack/neutron/neutron/services/service_base.py:49}} Jun 09 00:51:03.441861 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: > process after_init 55550001, True {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.441978 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Successfully loaded L3_ROUTER_NAT plugin. Description: L3 Router Service Plugin for basic L3 forwarding using OVN {{(pid=88492) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 09 00:51:03.442046 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading Plugin: auto_allocate Jun 09 00:51:03.442897 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: network precommit_update 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.442992 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron_lib.callbacks.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Subscribe: network precommit_create 55550000, False {{(pid=88492) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.443156 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Successfully loaded auto-allocated-topology plugin. Description: Auto Allocated Topology - aka get me a network. {{(pid=88492) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 09 00:51:03.443256 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading Plugin: tag Jun 09 00:51:03.445489 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Successfully loaded TAG plugin. Description: Tag support {{(pid=88492) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 09 00:51:03.445576 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading Plugin: timestamp Jun 09 00:51:03.446611 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Successfully loaded timestamp plugin. Description: Adds timestamps to Neutron resources with standard attributes {{(pid=88492) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 09 00:51:03.446693 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading Plugin: network_ip_availability Jun 09 00:51:03.447976 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Successfully loaded network-ip-availability plugin. Description: Provides IP availability data for each network and subnet. {{(pid=88492) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 09 00:51:03.448060 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading Plugin: flavors Jun 09 00:51:03.448751 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Successfully loaded FLAVORS plugin. Description: Neutron Flavors and Service Profiles manager plugin {{(pid=88492) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 09 00:51:03.448831 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading Plugin: revisions Jun 09 00:51:03.449673 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.manager [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Successfully loaded revision_plugin plugin. Description: Adds revision numbers to resources. {{(pid=88492) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 09 00:51:03.449782 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG oslo_concurrency.lockutils [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Lock "manager" "released" by "neutron.manager.NeutronManager._create_instance" :: held 1.485s {{(pid=88492) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} Jun 09 00:51:03.449965 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] get_extension_paths = OrderedDict({'/opt/stack/neutron/neutron/extensions': 1}) {{(pid=88492) get_extensions_path /opt/stack/neutron/neutron/api/extensions.py:640}} Jun 09 00:51:03.450048 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Initializing extension manager. Jun 09 00:51:03.450281 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: __init__.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.450364 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: __pycache__ {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.450448 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: aap_reject_multicast.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.450670 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Allowed address pairs reject multicast extension" alias="aap-reject-multicast" description="Reject multicast addresses in Allowed Address Pairs" updated="2026-04-01T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.450867 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension aap-reject-multicast not supported by any of loaded plugins Jun 09 00:51:03.450940 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: address_group.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.451215 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Address group" alias="address-group" description="Support address group" updated="2020-07-15T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.451378 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: address-group Jun 09 00:51:03.451459 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: address_scope.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.451694 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Address scope" alias="address-scope" description="Address scopes extension." updated="2015-07-26T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.451825 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: address-scope Jun 09 00:51:03.451894 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: admin_state_down_before_update.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.452083 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Enforce Router's Admin State Down Before Update Extension" alias="router-admin-state-down-before-update" description="Ensure that the admin state of a router is DOWN (admin_state_up=False) before updating the distributed attribute" updated="2019-07-02T15:56:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.452225 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension router-admin-state-down-before-update not supported by any of loaded plugins Jun 09 00:51:03.452302 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: agent.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.452549 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="agent" alias="agent" description="The agent management extension." updated="2013-02-03T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.452667 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: agent Jun 09 00:51:03.452737 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: agent_resources_synced.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.452915 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Agent's Resource View Synced to Placement" alias="agent-resources-synced" description="Stores success/failure of last sync to Placement" updated="2018-12-19T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.453033 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension agent-resources-synced not supported by any of loaded plugins Jun 09 00:51:03.453102 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: agent_sort_key.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.453310 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Agents Sort Key" alias="agent_sort_key" description="Enabling the keyword sort_key for sorting in agents." updated="2013-02-03T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.453447 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension agent_sort_key not supported by any of loaded plugins Jun 09 00:51:03.453521 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: allowedaddresspairs.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.454120 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Allowed Address Pairs" alias="allowed-address-pairs" description="Provides allowed address pairs" updated="2013-07-23T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.454310 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: allowed-address-pairs Jun 09 00:51:03.454374 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: auto_allocated_topology.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.454593 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Auto Allocated Topology Services" alias="auto-allocated-topology" description="Auto Allocated Topology Services." updated="2016-01-01T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.454723 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: auto-allocated-topology Jun 09 00:51:03.454785 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: availability_zone.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.455281 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Availability Zone" alias="availability_zone" description="The availability zone extension." updated="2015-01-01T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.455441 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: availability_zone Jun 09 00:51:03.455503 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: availability_zone_filter.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.455703 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Availability Zone Filter Extension" alias="availability_zone_filter" description="Add filter parameters to AvailabilityZone resource" updated="2018-06-22T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.455832 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension availability_zone_filter not supported by any of loaded plugins Jun 09 00:51:03.455909 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: data_plane_status.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.456103 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port data plane status extension" alias="data-plane-status" description="Expose status of underlying data plane" updated="2017-01-24T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.456253 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension data-plane-status not supported by any of loaded plugins Jun 09 00:51:03.456329 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: default_subnetpools.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.456528 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Default Subnetpools" alias="default-subnetpools" description="Provides ability to mark and use a subnetpool as the default." updated="2016-02-18T18:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.456651 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: default-subnetpools Jun 09 00:51:03.456718 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: dhcpagentscheduler.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.457046 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="DHCP Agent Scheduler" alias="dhcp_agent_scheduler" description="Schedule networks among dhcp agents" updated="2013-02-07T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.457177 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: dhcp_agent_scheduler Jun 09 00:51:03.457270 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: dns.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.457478 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="DNS Integration" alias="dns-integration" description="Provides integration with DNS." updated="2015-08-15T18:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.457595 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: dns-integration Jun 09 00:51:03.457646 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: dns_domain_ports.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.457837 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="dns_domain for ports" alias="dns-domain-ports" description="Allows the DNS domain to be specified for a network port." updated="2017-04-24T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.457950 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: dns-domain-ports Jun 09 00:51:03.458025 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: dns_integration_domain_keywords.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.458256 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="DNS domain names with keywords allowed" alias="dns-integration-domain-keywords" description="Allows to use keywords like , , and as DNS domain name" updated="2020-06-15T18:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.458354 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: dns-integration-domain-keywords Jun 09 00:51:03.458450 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: dvr.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.458642 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Distributed Virtual Router" alias="dvr" description="Enables configuration of Distributed Virtual Routers." updated="2014-06-1T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.458753 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension dvr not supported by any of loaded plugins Jun 09 00:51:03.458830 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: ecmp_routes.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.459001 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Equal Cost Multipath Routing" alias="ecmp_routes" description="Allows traffic to reach thesame destination via multiple different links " updated="2020-11-26T18:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.459116 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension ecmp_routes not supported by any of loaded plugins Jun 09 00:51:03.459209 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: empty_string_filtering.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.459392 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Empty String Filtering Extension" alias="empty-string-filtering" description="Allow filtering by attributes with empty string value" updated="2018-05-01T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.459526 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: empty-string-filtering Jun 09 00:51:03.459614 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: evpn.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.459773 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="EVPN" alias="evpn" description="Router extension to support EVPN Type-5 route advertisement" updated="2026-04-28T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.459886 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension evpn not supported by any of loaded plugins Jun 09 00:51:03.459957 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: expose_l3_conntrack_helper.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.460129 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Expose CT target rules for conntrack helper" alias="expose-l3-conntrack-helper" description="Expose allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.460268 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension expose-l3-conntrack-helper not supported by any of loaded plugins Jun 09 00:51:03.460347 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: expose_port_forwarding_in_fip.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.460560 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Expose Floating IP port forwardings" alias="expose-port-forwarding-in-fip" description="Expose port forwarding fields in floating IP response" updated="2018-06-28T16:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.460681 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension expose-port-forwarding-in-fip not supported by any of loaded plugins Jun 09 00:51:03.460766 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: external_net.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.460953 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron external network" alias="external-net" description="Adds external network attribute to network resource." updated="2013-01-14T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.461048 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: external-net Jun 09 00:51:03.461136 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: extra_dhcp_opt.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.461406 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron Extra DHCP options" alias="extra_dhcp_opt" description="Extra options configuration for DHCP. For example PXE boot options to DHCP clients can be specified (e.g. tftp-server, server-ip-address, bootfile-name)" updated="2013-03-17T12:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.461558 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: extra_dhcp_opt Jun 09 00:51:03.461617 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: extraroute.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.461809 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron Extra Route" alias="extraroute" description="Extra routes configuration for L3 router" updated="2013-02-01T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.461971 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: extraroute Jun 09 00:51:03.462061 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: extraroute_atomic.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.462338 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Atomically add/remove extra routes" alias="extraroute-atomic" description="Edit extra routes of a router on server side by atomically adding/removing extra routes" updated="2019-07-10T00:00:00+00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.462500 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: extraroute-atomic Jun 09 00:51:03.462625 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: filter_validation.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.462914 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Filter parameters validation" alias="filter-validation" description="Provides validation on filter parameters." updated="2018-07-04T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.463037 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: filter-validation Jun 09 00:51:03.463113 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: fip_pf_description.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.463299 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Floating IP Port Forwarding new attribute description" alias="floating-ip-port-forwarding-description" description="Add a description field to Port Forwarding rules" updated="2019-11-01T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.463412 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension floating-ip-port-forwarding-description not supported by any of loaded plugins Jun 09 00:51:03.463501 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: fip_pf_detail.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.463670 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Floating IP port forwarding detail" alias="floating-ip-port-forwarding-detail" description="Allow list floating ip return more port forwarding data, include'id' and 'internal_port_id'" updated="2022-05-06T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.463782 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: floating-ip-port-forwarding-detail Jun 09 00:51:03.463856 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: fip_pf_port_range.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.464046 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Floating IP Port Forwarding support port ranges" alias="floating-ip-port-forwarding-port-ranges" description="Add support to port ranges" updated="2020-07-01T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.464160 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension floating-ip-port-forwarding-port-ranges not supported by any of loaded plugins Jun 09 00:51:03.464271 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: fip_port_details.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.464442 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Floating IP Port Details Extension" alias="fip-port-details" description="Add port_details attribute to Floating IP resource" updated="2018-04-09T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.464565 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: fip-port-details Jun 09 00:51:03.464614 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: flavors.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.464827 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron Service Flavors" alias="flavors" description="Flavor specification for Neutron advanced services." updated="2015-09-17T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.464942 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: flavors Jun 09 00:51:03.464998 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: floating_ip_port_forwarding.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.465292 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Floating IP Port Forwarding" alias="floating-ip-port-forwarding" description="Allow port forwarding from floating IP port to an internal port" updated="2018-01-19T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.465413 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension floating-ip-port-forwarding not supported by any of loaded plugins Jun 09 00:51:03.465511 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: floatingip_pools.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.465705 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Floating IP Pools Extension" alias="floatingip-pools" description="Provides a floating IP pools API." updated="2018-03-21T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.465821 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: floatingip-pools Jun 09 00:51:03.465898 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: ip_allocation.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.466063 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="IP Allocation" alias="ip_allocation" description="IP allocation extension." updated="2016-06-10T23:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.466177 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension ip_allocation not supported by any of loaded plugins Jun 09 00:51:03.466280 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: ip_substring_port_filtering.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.466462 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="IP address substring filtering" alias="ip-substring-filtering" description="Provides IP address substring filtering when listing ports" updated="2017-11-28T09:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.466580 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension ip-substring-filtering not supported by any of loaded plugins Jun 09 00:51:03.466630 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l2_adjacency.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.466817 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="L2 Adjacency" alias="l2_adjacency" description="Display L2 Adjacency for Neutron Networks." updated="2016-04-12T16:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.466929 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension l2_adjacency not supported by any of loaded plugins Jun 09 00:51:03.467002 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.467405 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron L3 Router" alias="router" description="Router abstraction for basic L3 forwarding between L2 Neutron networks and access to external networks via a NAT gateway." updated="2012-07-20T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.467547 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: router Jun 09 00:51:03.467606 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3_agent_scheduler_ha_chassis_priority.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.467802 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="L3 Agent Scheduler HA Chassis Priority" alias="l3-agent-scheduler-ha-chassis-priority" description="Exposes the HA_Chassis priority in the L3 Agent Scheduler API responses and allows setting priority when scheduling a router to a chassis." updated="2026-03-31T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.467913 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: l3-agent-scheduler-ha-chassis-priority Jun 09 00:51:03.467967 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3_conntrack_helper.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.468584 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="L3 Conntrack helper" alias="l3-conntrack-helper" description="Allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.468760 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension l3-conntrack-helper not supported by any of loaded plugins Jun 09 00:51:03.468895 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3_enable_default_route_bfd.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.469075 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Enable Default Route BFD" alias="enable-default-route-bfd" description="Enables configurable BFD behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.469201 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: enable-default-route-bfd Jun 09 00:51:03.469292 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3_enable_default_route_ecmp.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.469504 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Enable Default Route ECMP" alias="enable-default-route-ecmp" description="Enables configurable ECMP behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.469615 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: enable-default-route-ecmp Jun 09 00:51:03.469696 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3_ext_gw_mode.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.469869 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron L3 Configurable external gateway mode" alias="ext-gw-mode" description="Extension of the router abstraction for specifying whether SNAT should occur on the external gateway" updated="2013-03-28T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.469977 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: ext-gw-mode Jun 09 00:51:03.470056 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3_ext_ha_mode.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.470269 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="HA Router extension" alias="l3-ha" description="Adds HA capability to routers." updated="2014-04-26T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.470381 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: l3-ha Jun 09 00:51:03.470482 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3_ext_ndp_proxy.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.470632 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Router support to enable NDP proxy" alias="l3-ext-ndp-proxy" description="Router extension to enable ndp proxy" updated="2020-08-25T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.470748 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension l3-ext-ndp-proxy not supported by any of loaded plugins Jun 09 00:51:03.470824 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3_extra_gws.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.471144 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron L3 External Gateway Multihoming" alias="external-gateway-multihoming" description="Allow multiple external gateway ports per router" updated="2023-01-18T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.471305 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: external-gateway-multihoming Jun 09 00:51:03.471390 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3_flavors.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.471578 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Router Flavor Extension" alias="l3-flavors" description="Flavor support for routers." updated="2016-05-17T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.471694 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: l3-flavors Jun 09 00:51:03.471770 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3_ndp_proxy.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.472064 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="name" alias="l3-ndp-proxy" description="description" updated="2021-06-29T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.472155 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension l3-ndp-proxy not supported by any of loaded plugins Jun 09 00:51:03.472251 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3_port_ip_change_not_allowed.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.472402 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Prevent L3 router ports IP address change extension" alias="l3-port-ip-change-not-allowed" description="Prevent change of IP address for some L3 router ports" updated="2018-10-09T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.472531 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension l3-port-ip-change-not-allowed not supported by any of loaded plugins Jun 09 00:51:03.472585 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: l3agentscheduler.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.473143 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="L3 Agent Scheduler" alias="l3_agent_scheduler" description="Schedule routers among l3 agents" updated="2013-02-07T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.473285 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: l3_agent_scheduler Jun 09 00:51:03.473366 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: local_ip.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.473642 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Local IP" alias="local_ip" description="Support Node Local Virtual IP" updated="2021-07-26T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.473753 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension local_ip not supported by any of loaded plugins Jun 09 00:51:03.473835 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: logging.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.474065 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Logging API Extension" alias="logging" description="Provides a logging API for resources such as security group and firewall." updated="2017-01-01T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.474209 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension logging not supported by any of loaded plugins Jun 09 00:51:03.474301 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: metering.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.474573 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron Metering" alias="metering" description="Neutron Metering extension." updated="2013-06-12T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.474684 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension metering not supported by any of loaded plugins Jun 09 00:51:03.474767 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: metering_source_and_destination_fields.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.475004 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron Metering with source and destination filtering" alias="metering_source_and_destination_fields" description="Neutron Metering extension that enables the use of source and destination IP prefixes to create metering label rules." updated="2020-08-21T08:12:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.475164 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension metering_source_and_destination_fields not supported by any of loaded plugins Jun 09 00:51:03.475276 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: multiprovidernet.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.475500 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Multi Provider Network" alias="multi-provider" description="Expose mapping of virtual networks to multiple physical networks" updated="2013-06-27T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.475623 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: multi-provider Jun 09 00:51:03.475682 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: netmtu.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.475915 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Network MTU" alias="net-mtu" description="Provides MTU attribute for a network resource." updated="2015-03-25T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.476035 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: net-mtu Jun 09 00:51:03.476108 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: netmtu_writable.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.476349 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Network MTU (writable)" alias="net-mtu-writable" description="Provides a writable MTU attribute for a network resource." updated="2017-07-12T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.476508 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: net-mtu-writable Jun 09 00:51:03.476613 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: network_availability_zone.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.476852 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Network Availability Zone" alias="network_availability_zone" description="Availability zone support for network." updated="2015-01-01T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.476973 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: network_availability_zone Jun 09 00:51:03.477045 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: network_ha.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.477261 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Network HA creation flag" alias="network_ha" description="Network high availability creation flag." updated="2023-04-27T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.477410 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension network_ha not supported by any of loaded plugins Jun 09 00:51:03.477513 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: network_ip_availability.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.477716 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Network IP Availability" alias="network-ip-availability" description="Provides IP availability data for each network and subnet." updated="2015-09-24T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.477836 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: network-ip-availability Jun 09 00:51:03.477888 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: network_ip_availability_details.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.478086 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Network IP Availability Details Extension" alias="network-ip-availability-details" description="Add ip_availability_details attribute to network-ip-availabilities resource." updated="2025-10-31T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.478243 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: network-ip-availability-details Jun 09 00:51:03.478326 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: network_segment_range.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.478641 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron Network Segment Range" alias="network-segment-range" description="Provides support for the network segment range management" updated="2018-11-29T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.478786 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension network-segment-range not supported by any of loaded plugins Jun 09 00:51:03.478839 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: pagination.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.479038 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Pagination support" alias="pagination" description="Extension that indicates that pagination is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.479092 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: pagination Jun 09 00:51:03.479210 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: port_device_profile.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.479446 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port device profile" alias="port-device-profile" description="Expose the port device profile (Cyborg)" updated="2020-12-17T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.479605 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: port-device-profile Jun 09 00:51:03.479683 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: port_hardware_offload_type.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.479862 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port hardware offload" alias="port-hardware-offload-type" description="Expose port hardware offload extension" updated="2023-05-09T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.479969 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension port-hardware-offload-type not supported by any of loaded plugins Jun 09 00:51:03.480034 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: port_hint_ovs_tx_steering.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.480219 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port hint: Open vSwitch Tx steering" alias="port-hint-ovs-tx-steering" description="Enable Port hint: Open vSwitch Tx steering" updated="2023-01-01T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.480336 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension port-hint-ovs-tx-steering not supported by any of loaded plugins Jun 09 00:51:03.480400 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: port_hints.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.480579 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port hints" alias="port-hints" description="Backend specific port hints to allow backend specific tuning" updated="2023-01-01T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.480720 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension port-hints not supported by any of loaded plugins Jun 09 00:51:03.480796 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: port_mac_address_override.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.480996 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron Port MAC address override" alias="port-mac-override" description="Allow overriding the MAC address of a direct-physical Port via the active binding profile" updated="2022-03-18T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.481243 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension port-mac-override not supported by any of loaded plugins Jun 09 00:51:03.481243 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: port_mac_address_regenerate.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.481475 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron Port MAC address regenerate" alias="port-mac-address-regenerate" description="Network port MAC address regenerate" updated="2018-05-03T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.481597 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: port-mac-address-regenerate Jun 09 00:51:03.481673 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: port_numa_affinity_policy.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.481842 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port NUMA affinity policy" alias="port-numa-affinity-policy" description="Expose the port NUMA affinity policy" updated="2020-07-08T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.481958 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: port-numa-affinity-policy Jun 09 00:51:03.482011 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: port_numa_affinity_policy_socket.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.482229 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port NUMA affinity policy "socket"" alias="port-numa-affinity-policy-socket" description="Adds "socket" to the supported port NUMA affinity policies" updated="2024-02-09T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.482356 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: port-numa-affinity-policy-socket Jun 09 00:51:03.482425 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: port_resource_request.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.482615 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port Resource Request" alias="port-resource-request" description="Expose resource request to Port" updated="2018-05-08T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.482753 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension port-resource-request not supported by any of loaded plugins Jun 09 00:51:03.482809 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: port_resource_request_groups.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.483018 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port Resource Request Groups" alias="port-resource-request-groups" description="Support requesting multiple groups of resources and traits from the same RP subtree in resource_request" updated="2021-08-02T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.483199 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension port-resource-request-groups not supported by any of loaded plugins Jun 09 00:51:03.483286 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: port_trusted_vif.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.483494 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port trusted vif" alias="port-trusted-vif" description="Expose port 'trusted' attribute in the API" updated="2024-07-10T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.483614 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension port-trusted-vif not supported by any of loaded plugins Jun 09 00:51:03.483666 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: portbindings.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.483853 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port Binding" alias="binding" description="Expose port bindings of a virtual port to external application" updated="2014-02-03T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.483971 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: binding Jun 09 00:51:03.484048 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: portbindings_extended.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.484293 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port Bindings Extended" alias="binding-extended" description="Expose port bindings of a virtual port to external application" updated="2017-07-17T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.484407 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: binding-extended Jun 09 00:51:03.484471 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: portsecurity.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.484658 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port Security" alias="port-security" description="Provides port security" updated="2012-07-23T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.484792 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: port-security Jun 09 00:51:03.484874 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: project_id.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.485060 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="project_id field enabled" alias="project-id" description="Extension that indicates that project_id field is enabled." updated="2016-09-09T09:09:09-09:09" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.485158 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: project-id Jun 09 00:51:03.485282 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: providernet.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.485734 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Provider Network" alias="provider" description="Expose mapping of virtual networks to physical networks" updated="2012-09-07T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.485862 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: provider Jun 09 00:51:03.485927 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: pvlan.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.486121 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Private VLAN" alias="pvlan" description="The Private VLAN semantics extension." updated="2026-02-03T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.486248 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension pvlan not supported by any of loaded plugins Jun 09 00:51:03.486321 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qinq.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.486583 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="QinQ networks" alias="qinq" description="Expose network 'qinq' attribute in the API. This allows to configure network to allow vlan in vlan configuration using 0x88a8 ethertype (QinQ)." updated="2024-12-02T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.486779 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: qinq Jun 09 00:51:03.486884 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.487661 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Quality of Service" alias="qos" description="The Quality of Service extension." updated="2015-06-08T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.487783 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos not supported by any of loaded plugins Jun 09 00:51:03.487876 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_bw_limit_direction.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.488081 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Direction for QoS bandwidth limit rule" alias="qos-bw-limit-direction" description="Allow to configure QoS bandwidth limit rule with specific direction: ingress or egress" updated="2017-04-10T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.488286 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-bw-limit-direction not supported by any of loaded plugins Jun 09 00:51:03.488395 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_bw_minimum_ingress.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.488689 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Ingress direction for QoS minimum bandwidth rule" alias="qos-bw-minimum-ingress" description="Allow to configure QoS minumum bandwidth rule with ingress direction." updated="2018-07-09T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.488882 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-bw-minimum-ingress not supported by any of loaded plugins Jun 09 00:51:03.488958 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_default.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.489152 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="QoS default policy" alias="qos-default" description="Expose the QoS default policy per project" updated="2017-041-06T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.489322 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-default not supported by any of loaded plugins Jun 09 00:51:03.489407 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_fip.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.489645 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Floating IP QoS" alias="qos-fip" description="The floating IP Quality of Service extension" updated="2017-07-20T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.489840 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-fip not supported by any of loaded plugins Jun 09 00:51:03.489951 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_fip_network_policy.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.490241 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="QoS floating IP network policy ID" alias="qos-fip-network-policy" description="Adds a the QoS network ID to the floating IP definition" updated="2021-11-15T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.490391 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-fip-network-policy not supported by any of loaded plugins Jun 09 00:51:03.490478 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_gateway_ip.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.490661 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Router gateway IP QoS" alias="qos-gateway-ip" description="The Router gateway IP Quality of Service extension" updated="2018-02-24T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.490817 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-gateway-ip not supported by any of loaded plugins Jun 09 00:51:03.490928 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_port_network_policy.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.491204 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="QoS port network policy ID" alias="qos-port-network-policy" description="Adds a the QoS network ID to the port definition" updated="2019-11-01T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.491408 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-port-network-policy not supported by any of loaded plugins Jun 09 00:51:03.491520 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_pps_minimum_rule.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.491791 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="QoS minimum packet rate rule" alias="qos-pps-minimum" description="Allow to configure QoS minimum packet rate rule." updated="2021-07-14T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.491975 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-pps-minimum not supported by any of loaded plugins Jun 09 00:51:03.492084 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_pps_minimum_rule_alias.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.492379 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="QoS minimum packet rate rule alias" alias="qos-pps-minimum-rule-alias" description="API to enable GET, PUT and DELETE operations on QoS minimum packet rate rule without specifying policy ID" updated="2021-10-22T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.492577 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-pps-minimum-rule-alias not supported by any of loaded plugins Jun 09 00:51:03.492684 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_pps_rule.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.492976 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="QoS Rule Type Packet per Second Extension" alias="qos-pps" description="Add QoS Rule Type Packet per Second" updated="2021-05-12T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.493161 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-pps not supported by any of loaded plugins Jun 09 00:51:03.493296 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_rule_type_details.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.493588 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Details of QoS rule types" alias="qos-rule-type-details" description="Expose details about QoS rule types supported by loaded backend drivers" updated="2017-06-22T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.493786 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-rule-type-details not supported by any of loaded plugins Jun 09 00:51:03.493910 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_rule_type_filter.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.494182 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Allow to filter the list of QoS rule types" alias="qos-rule-type-filter" description="Allows to filter the QoS rule type list adding two new flags. "all_rules" prints all implemented QoS rule types."all_supported" prints all supported QoS rule types by the loaded mechanism drivers" updated="2022-02-02T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.494369 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-rule-type-filter not supported by any of loaded plugins Jun 09 00:51:03.494471 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: qos_rules_alias.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.494732 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Quality of Service rules alias API" alias="qos-rules-alias" description="API to enable GET, PUT and DELETE operations on QoS policy rules without specifying policy ID" updated="2018-10-07T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.494918 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension qos-rules-alias not supported by any of loaded plugins Jun 09 00:51:03.495031 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: quota_check_limit.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.495314 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Quota engine limit check" alias="quota-check-limit" description="Support for checking the resource usage before applying a new quota limit" updated="2021-09-08T16:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.495525 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: quota-check-limit Jun 09 00:51:03.495635 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: quota_check_limit_default.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.496041 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Quota engine limit check by default" alias="quota-check-limit-default" description="By default, the Neutron quota engine checks the resource usage before applying a new quota limit" updated="2024-08-21T16:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.496105 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: quota-check-limit-default Jun 09 00:51:03.496229 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: quotasv2.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.496845 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Quota management support" alias="quotas" description="Expose functions for quotas management per project" updated="2012-07-29T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.496960 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: quotas Jun 09 00:51:03.497081 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: quotasv2_detail.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.497514 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Quota details management support" alias="quota_details" description="Expose functions for quotas usage statistics per project" updated="2017-02-10T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.497635 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: quota_details Jun 09 00:51:03.497762 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: rbac.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.498376 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="RBAC Policies" alias="rbac-policies" description="Allows creation and modification of policies that control project access to resources." updated="2015-06-17T12:15:12-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.498600 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: rbac-policies Jun 09 00:51:03.498705 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: rbac_address_group.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.498989 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Add address_group type to RBAC" alias="rbac-address-group" description="Add address_group type to network RBAC" updated="2021-01-20T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.499179 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension rbac-address-group not supported by any of loaded plugins Jun 09 00:51:03.499301 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: rbac_address_scope.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.499591 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Add address_scope type to RBAC" alias="rbac-address-scope" description="Add address_scope type to RBAC" updated="2020-02-12T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.499798 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: rbac-address-scope Jun 09 00:51:03.499911 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: rbac_security_groups.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.500200 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Add security_group type to network RBAC" alias="rbac-security-groups" description="Add security_group type to network RBAC" updated="2019-02-14T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.500434 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: rbac-security-groups Jun 09 00:51:03.500556 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: rbac_subnetpool.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.500830 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Add subnetpool type to RBAC" alias="rbac-subnetpool" description="Add subnetpool type to RBAC" updated="2020-02-05T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.501032 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension rbac-subnetpool not supported by any of loaded plugins Jun 09 00:51:03.501143 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: revisionifmatch.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.501545 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="If-Match constraints based on revision_number" alias="revision-if-match" description="Extension indicating that If-Match based on revision_number is supported." updated="2016-12-11T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.501650 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: revision-if-match Jun 09 00:51:03.501753 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: revisions.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.502039 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Resource revision numbers" alias="standard-attr-revisions" description="This extension will display the revision number of neutron resources." updated="2016-04-11T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.502226 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: standard-attr-revisions Jun 09 00:51:03.502342 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: router_availability_zone.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.502578 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Router Availability Zone" alias="router_availability_zone" description="Availability zone support for router." updated="2015-01-01T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.502739 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: router_availability_zone Jun 09 00:51:03.502838 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: router_enable_snat.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.503063 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Exposes the router "enable_snat" read-only configuration flag" alias="router-enable-snat" description="Exposes the router "enable_snat" read-only configuration flag" updated="2025-09-18T18:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.503247 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: router-enable-snat Jun 09 00:51:03.503339 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: routerservicetype.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.503558 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Router Service Type" alias="router-service-type" description="Provides router service type" updated="2013-01-29T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.503715 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension router-service-type not supported by any of loaded plugins Jun 09 00:51:03.503780 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: security_groups_default_rules.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.504178 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Default rules for security groups" alias="security-groups-default-rules" description="Configure set of security group rules used as default rules for every new security group" updated="2022-12-19T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.504345 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: security-groups-default-rules Jun 09 00:51:03.504476 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: security_groups_normalized_cidr.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.504603 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Normalized CIDR field for security group rules" alias="security-groups-normalized-cidr" description="Add new field with normalized remote_ip_prefix cidr in SG rule" updated="2020-07-28T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.504768 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: security-groups-normalized-cidr Jun 09 00:51:03.504864 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: security_groups_port_filtering.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.505120 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Port filtering on security groups" alias="port-security-groups-filtering" description="Provides security groups filtering when listing ports" updated="2018-01-09T09:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.505328 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension port-security-groups-filtering not supported by any of loaded plugins Jun 09 00:51:03.505442 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: security_groups_remote_address_group.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.505710 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Remote address group id field for security group rules" alias="security-groups-remote-address-group" description="Add new field of remote address group id in SG rules" updated="2020-08-25T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.505883 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: security-groups-remote-address-group Jun 09 00:51:03.505985 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: security_groups_rules_belongs_to_default_sg.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.506272 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Security group rule belongs to the project's default security group" alias="security-groups-rules-belongs-to-default-sg" description="Flag to determine if the security group rule belongs to the project's default security group" updated="2023-05-23T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.506469 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: security-groups-rules-belongs-to-default-sg Jun 09 00:51:03.506566 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: security_groups_shared_filtering.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.507067 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Security group filtering on the shared field" alias="security-groups-shared-filtering" description="Support filtering security groups on the shared field" updated="2021-10-05T09:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.507280 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: security-groups-shared-filtering Jun 09 00:51:03.507387 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: securitygroup.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.509290 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="security-group" alias="security-group" description="The security groups extension." updated="2012-10-05T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.509462 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: security-group Jun 09 00:51:03.509556 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: segment.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.509871 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Segment" alias="segment" description="Segments extension." updated="2016-02-24T17:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.510072 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension segment not supported by any of loaded plugins Jun 09 00:51:03.510182 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: segments_peer_subnet_host_routes.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.510495 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Segments peer-subnet host routes" alias="segments-peer-subnet-host-routes" description="Add host routes to subnets on a routed network (segments)" updated="2018-06-12T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.510684 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension segments-peer-subnet-host-routes not supported by any of loaded plugins Jun 09 00:51:03.510794 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: servicetype.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.511101 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Neutron Service Type Management" alias="service-type" description="API for retrieving service providers for Neutron advanced services" updated="2013-01-20T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.511321 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: service-type Jun 09 00:51:03.511420 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: sorting.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.511621 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Sorting support" alias="sorting" description="Extension that indicates that sorting is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.511780 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: sorting Jun 09 00:51:03.511833 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: standard_attr_segment.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.512019 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Standard Attribute Segment Extension" alias="standard-attr-segment" description="Add standard attributes to Segment resource" updated="2018-04-09T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.512206 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension standard-attr-segment not supported by any of loaded plugins Jun 09 00:51:03.512327 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: standardattrdescription.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.512631 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="standard-attr-description" alias="standard-attr-description" description="Extension to add descriptions to standard attributes" updated="2016-02-10T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.512801 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: standard-attr-description Jun 09 00:51:03.512868 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: stateful_security_group.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.513117 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Stateful security group" alias="stateful-security-group" description="Indicates if the security group is stateful or not" updated="2019-11-26T09:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.513253 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: stateful-security-group Jun 09 00:51:03.513314 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: stdattrs_common.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.513556 np0000002381 neutron-ovn-maintenance-worker[88492]: WARNING neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Did not find expected name "Stdattrs_common" in /opt/stack/neutron/neutron/extensions/stdattrs_common.py Jun 09 00:51:03.513630 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: subnet_dns_publish_fixed_ip.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.513866 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Subnet DNS publish fixed IP" alias="subnet-dns-publish-fixed-ip" description="Support choosing to publish DNS records for IPs from a subnet" updated="2019-05-24T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.513995 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: subnet-dns-publish-fixed-ip Jun 09 00:51:03.514092 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: subnet_external_network.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.514267 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Subnet belongs to an external network" alias="subnet-external-network" description="Informs if the subnet belongs to an external network" updated="2024-02-05T18:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.514385 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: subnet-external-network Jun 09 00:51:03.514467 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: subnet_onboard.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.514665 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Subnet Onboard" alias="subnet_onboard" description="Provides support for onboarding subnets into subnet pools" updated="2018-12-18T09:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.514784 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: subnet_onboard Jun 09 00:51:03.514858 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: subnet_segmentid_writable.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.515063 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Subnet SegmentID (writable)" alias="subnet-segmentid-writable" description="Provides a writable segment_id attribute for a subnet resource." updated="2018-03-12T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.515183 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension subnet-segmentid-writable not supported by any of loaded plugins Jun 09 00:51:03.515275 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: subnet_service_types.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.515491 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Subnet service types" alias="subnet-service-types" description="Provides ability to set the subnet service_types field" updated="2016-03-15T18:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.515618 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: subnet-service-types Jun 09 00:51:03.515686 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: subnetallocation.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.515898 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Subnet Allocation" alias="subnet_allocation" description="Enables allocation of subnets from a subnet pool" updated="2015-03-30T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.516076 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: subnet_allocation Jun 09 00:51:03.516162 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: subnetpool_prefix_ops.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.516394 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Subnet Pool Prefix Operations" alias="subnetpool-prefix-ops" description="Provides support for adjusting the prefix list of subnet pools" updated="2019-02-08T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.516526 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: subnetpool-prefix-ops Jun 09 00:51:03.516605 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: tag_creation.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.516771 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Tag creation extension" alias="tag-creation" description="Allow to create multiple tags for a resource" updated="2024-23-07T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.516915 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: tag-creation Jun 09 00:51:03.516994 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: tag_ports_during_bulk_creation.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.517227 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Tag Ports During Bulk Creation" alias="tag-ports-during-bulk-creation" description="Allow to tag ports during bulk creation" updated="2019-12-29T19:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.517376 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension tag-ports-during-bulk-creation not supported by any of loaded plugins Jun 09 00:51:03.517454 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: tagging.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.518157 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Tag support for resources with standard attribute: port, subnet, subnetpool, network, security_group, router, floatingip, policy, trunk, network_segment_range" alias="standard-attr-tag" description="Enables to set tag on resources with standard attribute." updated="2017-01-01T00:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.518308 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: standard-attr-tag Jun 09 00:51:03.518495 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: timestamp.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.518674 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Resource timestamps" alias="standard-attr-timestamp" description="Adds created_at and updated_at fields to all Neutron resources that have Neutron standard attributes." updated="2016-09-12T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.518841 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: standard-attr-timestamp Jun 09 00:51:03.518927 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: trunk.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.519149 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Trunk Extension" alias="trunk" description="Provides support for trunk ports" updated="2016-01-01T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.519306 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension trunk not supported by any of loaded plugins Jun 09 00:51:03.519390 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: trunk_details.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.519596 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Trunk port details" alias="trunk-details" description="Expose trunk port details" updated="2016-01-01T10:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.519706 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension trunk-details not supported by any of loaded plugins Jun 09 00:51:03.519772 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: uplink_status_propagation.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.519958 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Uplink status propagation extension" alias="uplink-status-propagation" description="Provides ability to propagate port uplink status." updated="2018-05-31T18:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.520072 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension uplink-status-propagation not supported by any of loaded plugins Jun 09 00:51:03.520140 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: uplink_status_propagation_updatable.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.520335 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Uplink status propagation updatable extension" alias="uplink-status-propagation-updatable" description="Provides ability to update the port uplink status." updated="2024-09-03T18:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.520463 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Extension uplink-status-propagation-updatable not supported by any of loaded plugins Jun 09 00:51:03.520528 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loading extension file: vlantransparent.py {{(pid=88492) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 09 00:51:03.520936 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Ext name="Vlantransparent" alias="vlan-transparent" description="Provides Vlan Transparent Networks" updated="2015-03-23T09:00:00-00:00" {{(pid=88492) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 09 00:51:03.521061 np0000002381 neutron-ovn-maintenance-worker[88492]: INFO neutron.api.extensions [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Loaded extension: vlan-transparent Jun 09 00:51:03.525065 np0000002381 neutron-ovn-maintenance-worker[88492]: WARNING oslo_service.backend._threading.service [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Service MaintenanceWorker is not picklable with spawn; falling back to fork. Please make the service spawn-safe to avoid this fallback. Jun 09 00:51:03.525454 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Full set of CONF: {{(pid=88492) _load_service_manager_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:59}} Jun 09 00:51:03.525521 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ******************************************************************************** {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3229}} Jun 09 00:51:03.525607 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Configuration options gathered from: {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3230}} Jun 09 00:51:03.525652 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3231}} Jun 09 00:51:03.525748 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3232}} Jun 09 00:51:03.525800 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ================================================================================ {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3237}} Jun 09 00:51:03.525919 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] agent_down_time = 75 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.526000 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] allow_automatic_dhcp_failover = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.526082 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] allow_bulk = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.526410 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.526488 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] api_extensions_path = {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.526564 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] api_paste_config = api-paste.ini {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.526644 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] api_workers = 2 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.526821 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] auth_strategy = keystone {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.526821 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] base_mac = fa:16:3e:00:00:00 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.526886 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] client_socket_timeout = 900 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.526971 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] config_dir = [] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.527034 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.527122 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] config_source = [] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.527245 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] control_exchange = neutron {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.527351 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] core_plugin = ml2 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.527457 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] debug = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.527545 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] default_availability_zones = [] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.527694 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.527780 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] dhcp_agent_notification = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.527866 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] dhcp_agents_per_network = 1 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.527938 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] dhcp_lease_duration = 86400 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528017 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] dhcp_load_type = networks {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528096 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] dns_domain = openstacklocal {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528182 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528283 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] enable_default_route_bfd = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528357 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] enable_default_route_ecmp = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528448 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] enable_dvr = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528524 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] enable_new_agents = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528599 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] enable_services_on_agents_with_admin_state_down = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528680 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] enable_snat_by_default = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528745 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] enable_traditional_dhcp = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528839 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] external_dns_driver = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528889 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] filter_validation = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.528985 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] global_physnet_mtu = 1430 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.529069 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] graceful_shutdown_timeout = 60 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.529233 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] host = np0000002381 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.529343 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] host_dvr_for_dhcp = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.529452 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] http_retries = 3 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.529531 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] instance_format = [instance: %(uuid)s] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.529617 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.529700 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ipam_driver = internal {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.529763 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ipv6_pd_enabled = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.529847 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] l3_ha = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.529914 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530002 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] l3_ha_network_physical_name = {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530076 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] l3_ha_network_type = {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530162 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] log_color = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530260 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] log_config_append = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530344 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530438 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] log_dir = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530518 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] log_file = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530583 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] log_options = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530640 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] log_rotate_interval = 1 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530726 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] log_rotate_interval_type = days {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530833 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] log_rotation_type = none {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530908 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.530970 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.531049 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.531134 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.531204 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.531319 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] max_allowed_address_pair = 10 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.531412 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] max_dns_nameservers = 5 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.531512 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] max_header_line = 16384 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.531593 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] max_l3_agents_per_router = 3 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.531673 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] max_logfile_count = 30 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.531737 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] max_logfile_size_mb = 200 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.531820 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] max_routes = 30 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.531896 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] max_subnet_host_routes = 20 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.532056 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] my_ip = 10.4.3.231 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.532056 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] my_ipv6 = ::1 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.532128 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] network_auto_schedule = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.532226 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] network_link_prefix = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.532312 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.532395 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] notify_nova_on_port_data_changes = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.532488 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] notify_nova_on_port_status_changes = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.532605 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] pagination_max_limit = -1 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.532689 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] periodic_fuzzy_delay = 5 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.532769 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] periodic_interval = 40 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.532843 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] publish_errors = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.532918 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] rate_limit_burst = 0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.533019 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] rate_limit_except_level = CRITICAL {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.533096 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] rate_limit_interval = 0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.533180 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] router_distributed = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.533322 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] rpc_resources_processing_step = 20 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.533458 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] rpc_response_max_timeout = 600 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.533587 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] rpc_response_timeout = 60 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.533721 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] rpc_state_report_workers = 0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.533844 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] rpc_workers = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.533965 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] send_events_interval = 2 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.534110 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] service_plugins = ['ovn-router'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.534259 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] setproctitle = on {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.534393 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] shell_completion = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.534538 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] state_path = /opt/stack/data/neutron {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.534669 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] syslog_log_facility = LOG_USER {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.534805 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] tcp_keepidle = 600 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.534946 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] transport_url = **** {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.535081 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] use_journal = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.535228 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] use_json = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.535365 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] use_stderr = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.535513 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] use_syslog = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.535644 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] vlan_qinq = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.535768 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] vlan_transparent = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.535899 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] wsgi_default_pool_size = 100 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.536029 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] wsgi_keep_alive = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.536167 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.536321 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] wsgi_server_debug = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.536494 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_concurrency.disable_process_locking = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.536640 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.536791 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler.connection_string = messaging:// {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.536932 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler.enabled = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.537073 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler.es_doc_type = notification {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.537234 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler.es_scroll_size = 10000 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.537382 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler.es_scroll_time = 2m {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.537533 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler.filter_error_trace = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.537671 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler.hmac_keys = **** {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.537811 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler.sentinel_service_name = mymaster {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.537963 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler.socket_timeout = 0.1 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.538086 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler.trace_requests = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.538240 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler.trace_sqlalchemy = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.538388 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler_jaeger.process_tags = {} {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.538533 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler_jaeger.service_name_prefix = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.538665 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] profiler_otlp.service_name_prefix = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.538804 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.538936 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.539074 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_metrics.metrics_process_name = {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.539230 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.539364 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.539524 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_policy.enforce_new_defaults = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.539833 np0000002381 neutron-ovn-maintenance-worker[88492]: WARNING oslo_config.cfg [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Deprecated: Option "enforce_scope" from group "oslo_policy" is deprecated for removal (This configuration was added temporarily to facilitate a smooth transition to the new RBAC. OpenStack will always enforce scope checks. This configuration option is deprecated and will be removed in the 2025.2 cycle.). Its value may be silently ignored in the future. Jun 09 00:51:03.539990 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_policy.enforce_scope = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.540128 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_policy.policy_default_rule = default {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.540302 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.540457 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.540598 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.540728 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.540859 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.540992 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.541117 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.541285 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_policy.remote_timeout = 60.0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.541441 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.541585 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] service_providers.service_provider = [] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.541723 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.auth_section = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.541860 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.auth_type = password {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.541995 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.auth_url = https://10.4.3.231/identity {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.542184 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.cafile = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.542283 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.certfile = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.542406 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.collect_timing = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.542543 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.default_domain_id = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.542667 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.default_domain_name = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.542796 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.domain_id = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.542917 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.domain_name = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.543048 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.endpoint_type = public {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.543167 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.insecure = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.543310 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.keyfile = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.543454 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.password = **** {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.543571 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.project_domain_id = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.543683 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.project_domain_name = Default {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.543802 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.project_id = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.543928 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.project_name = service {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.544050 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.region_name = RegionOne {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.544173 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.split_loggers = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.544335 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.system_scope = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.544463 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.timeout = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.544594 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.trust_id = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.544733 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.user_domain_id = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.544872 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.user_domain_name = Default {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.545005 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.user_id = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.545153 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] nova.username = nova {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.545321 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] placement.auth_section = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.545472 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] placement.auth_type = password {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.545608 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] placement.cafile = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.545729 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] placement.certfile = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.545861 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] placement.collect_timing = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.546004 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] placement.endpoint_type = public {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.546128 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] placement.insecure = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.546265 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] placement.keyfile = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.546381 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] placement.region_name = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.546518 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] placement.split_loggers = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.546630 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] placement.timeout = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.546763 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.auth_section = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.546892 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.auth_type = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.547019 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.cafile = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.547150 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.certfile = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.547303 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.collect_timing = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.547457 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.connect_retries = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.547590 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.connect_retry_delay = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.547711 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.enable_notifications = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.547836 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.endpoint_override = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.547965 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.insecure = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.548092 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.interface = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.548229 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.keyfile = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.548359 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.max_version = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.548499 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.min_version = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.548630 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.region_name = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.548757 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.retriable_status_codes = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.548915 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.service_name = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.549048 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.service_type = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.549201 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.split_loggers = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.549351 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.status_code_retries = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.549506 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.status_code_retry_delay = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.549645 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.timeout = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.549782 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.valid_interfaces = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.549921 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ironic.version = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.550076 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] cli_script.dry_run = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.550245 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.550384 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.550512 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.550622 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.550735 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.550853 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.550987 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.551164 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.551302 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.hostname = np0000002381 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.551455 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.551618 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.551761 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.551906 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.552045 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.552203 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.552354 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.552501 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.552644 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.552804 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.552940 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.553068 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.553222 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.553330 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.553428 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.553557 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.553693 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.553826 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.553968 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.554107 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.554264 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.ssl = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.554545 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.554545 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.554670 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.554795 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.ssl_enforce_hostname_verification = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.554938 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.555082 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.ssl_version = {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.555243 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_rabbit.use_queue_manager = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.555405 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_notifications.driver = [] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.555544 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_notifications.retry = -1 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.555700 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.555832 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_messaging_notifications.transport_url = **** {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.555977 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_reports.file_event_handler = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.556118 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_reports.file_event_handler_interval = 1 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.556277 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_reports.log_dir = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.556439 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.556586 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] QUOTAS.default_quota = -1 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.556737 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.556885 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] QUOTAS.quota_floatingip = 50 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.557013 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] QUOTAS.quota_network = 100 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.557150 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] QUOTAS.quota_port = 500 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.557315 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] QUOTAS.quota_rbac_policy = 10 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.557464 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] QUOTAS.quota_router = 10 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.557605 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] QUOTAS.quota_security_group = 10 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.557738 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] QUOTAS.quota_security_group_rule = 100 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.557873 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] QUOTAS.quota_subnet = 100 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.558022 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] QUOTAS.track_quota_usage = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.558143 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.558294 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep.group = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.558447 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep.helper_command = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.558552 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep.log_daemon_traceback = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.558643 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep.logger_name = oslo_privsep.daemon {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.558728 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep.thread_pool_size = 12 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.558807 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep.user = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.558893 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_dhcp_release.capabilities = [21, 12] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.558977 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_dhcp_release.group = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.559055 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_dhcp_release.helper_command = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.559131 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_dhcp_release.log_daemon_traceback = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.559233 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.559315 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_dhcp_release.thread_pool_size = 12 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.559391 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_dhcp_release.user = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.559493 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.559573 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_ovs_vsctl.group = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.559649 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_ovs_vsctl.helper_command = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.559726 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.559809 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.559887 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_ovs_vsctl.thread_pool_size = 12 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.559970 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_ovs_vsctl.user = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.560083 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_namespace.capabilities = [21] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.560215 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_namespace.group = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.560354 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_namespace.helper_command = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.560506 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_namespace.log_daemon_traceback = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.560650 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.560789 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_namespace.thread_pool_size = 12 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.560928 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_namespace.user = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.561075 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_conntrack.capabilities = [12] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.561610 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_conntrack.group = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.561610 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_conntrack.helper_command = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.561610 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_conntrack.log_daemon_traceback = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.561780 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.561865 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_conntrack.thread_pool_size = 12 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.561999 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_conntrack.user = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.562143 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_link.capabilities = [12, 21] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.562306 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_link.group = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.562479 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_link.helper_command = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.562566 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_link.log_daemon_traceback = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.562653 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_link.logger_name = oslo_privsep.daemon {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.562735 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_link.thread_pool_size = 12 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.562816 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] privsep_link.user = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.562911 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.562993 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.563073 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] AGENT.use_helper_for_ns_read = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.563157 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] SECURITYGROUP.enable_ipset = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.563264 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] SECURITYGROUP.enable_security_group = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.563344 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] SECURITYGROUP.firewall_driver = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.563440 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] SECURITYGROUP.permitted_ethertypes = [] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.563530 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] experimental.ipv6_pd_enabled = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.563630 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.563709 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2.external_network_type = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.563795 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2.mechanism_drivers = ['ovn'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.563882 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2.overlay_ip_version = 4 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.563967 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2.path_mtu = 0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.564044 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2.physical_network_mtus = {} {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.564132 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2.project_network_types = ['geneve'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.564236 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.564328 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.564428 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2_type_flat.flat_networks = ['public'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.564520 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2_type_geneve.max_header_size = 38 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.564604 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.564692 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.564822 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] network_log.burst_limit = 25 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.564884 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] network_log.local_output_log_base = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.564927 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] network_log.rate_limit = 100 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565001 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.asyncio_connection = **** {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565074 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.asyncio_slave_connection = **** {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565153 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.backend = sqlalchemy {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565250 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.connection = **** {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565331 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.connection_debug = 0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565421 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.connection_parameters = {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565502 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.connection_recycle_time = 3600 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565580 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.connection_trace = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565664 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.db_inc_retry_interval = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565767 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.db_max_retries = 20 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565810 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.db_max_retry_interval = 10 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565875 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.db_retry_interval = 1 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.565951 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.max_overflow = 50 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566030 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.max_pool_size = 5 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566100 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.max_retries = 10 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566181 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.mysql_sql_mode = TRADITIONAL {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566278 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.mysql_wsrep_sync_wait = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566354 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.pool_timeout = None {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566440 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.retry_interval = 10 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566516 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.slave_connection = **** {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566597 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.sqlite_synchronous = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566669 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.synchronous_reader = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566744 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] database.use_db_reconnect = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566833 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.bfd_min_rx = 1000 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566909 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.bfd_min_tx = 100 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.566985 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.bfd_mult = 3 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.567061 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.broadcast_arps_to_all_routers = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.567138 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.dhcp_default_lease_time = 43200 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.567232 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.567312 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.dns_records_ovn_owned = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.567392 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.dns_servers = [] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.567487 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.enable_distributed_floating_ip = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.567565 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.fdb_age_threshold = 300 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.567651 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ha_failover_strategy = normal {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.567739 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.live_migration_activation_strategy = rarp {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.567816 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.localnet_learn_fdb = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.567894 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.mac_binding_age_threshold = 300 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.567983 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.neutron_sync_mode = log {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.568057 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_dhcp4_global_options = {} {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.568129 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_dhcp6_global_options = {} {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.568226 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_l3_scheduler = leastloaded {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.568314 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_metadata_enabled = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.568396 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.568488 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.568578 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_nb_connection = ['ssl:10.4.3.231:6641'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.568660 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.568738 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_router_indirect_snat = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.568844 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.568960 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.569077 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_sb_connection = ['ssl:10.4.3.231:6642'] {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.569169 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.569285 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovs_create_tap = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.569346 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovsdb_connection_timeout = 180 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.569450 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovsdb_log_level = INFO {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.569533 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovsdb_probe_interval = 60000 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.569616 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.ovsdb_retry_max_interval = 180 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.569689 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.stateless_nat_enabled = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.569768 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.569855 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] OVS.bridge_mac_table_size = 50000 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.569935 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] OVS.igmp_flood = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.570021 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] OVS.igmp_flood_reports = True {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.570084 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] OVS.igmp_flood_unregistered = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.570171 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] OVS.igmp_snooping_enable = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.570267 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] OVS.ovsdb_timeout = 10 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.570354 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn_nb_global.fdb_removal_limit = 0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.570452 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn_nb_global.ignore_lsp_down = False {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.570548 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.570617 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG cotyledon.oslo_config_glue [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] ******************************************************************************** {{(pid=88492) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3263}} Jun 09 00:51:03.570707 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG oslo_service.backend._threading.service [None req-a72bb24c-e896-4304-a72f-74f1084060fe None None] Selected the multiprocessing context: and updated it in Cotyledon ServiceManager: {{(pid=88492) launch_service /opt/stack/data/venv/lib/python3.12/site-packages/oslo_service/backend/_threading/service.py:405}} Jun 09 00:51:03.590522 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] Full set of CONF: {{(pid=89255) _load_service_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:65}} Jun 09 00:51:03.590983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3229}} Jun 09 00:51:03.590983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] Configuration options gathered from: {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3230}} Jun 09 00:51:03.590983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3231}} Jun 09 00:51:03.590983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3232}} Jun 09 00:51:03.590983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ================================================================================ {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3237}} Jun 09 00:51:03.591209 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] agent_down_time = 75 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.591209 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] allow_automatic_dhcp_failover = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.591379 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] allow_bulk = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.591573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.591666 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] api_extensions_path = {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.591750 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] api_paste_config = api-paste.ini {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.591836 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] api_workers = 2 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.591918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] auth_strategy = keystone {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.591995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] base_mac = fa:16:3e:00:00:00 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.592077 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] client_socket_timeout = 900 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.592203 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] config_dir = [] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.592266 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.592339 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] config_source = [] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.592497 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] control_exchange = neutron {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.592497 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] core_plugin = ml2 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.592599 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] debug = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.592635 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] default_availability_zones = [] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.592790 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.592896 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agent_notification = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.592936 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agents_per_network = 1 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593111 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] dhcp_lease_duration = 86400 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593111 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] dhcp_load_type = networks {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] dns_domain = openstacklocal {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593440 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_bfd = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593440 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_ecmp = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593440 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] enable_dvr = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593577 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] enable_new_agents = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593577 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] enable_services_on_agents_with_admin_state_down = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593645 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] enable_snat_by_default = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593717 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] enable_traditional_dhcp = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593786 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] external_dns_driver = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593786 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] fatal_deprecations = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] filter_validation = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.593914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] global_physnet_mtu = 1430 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.594037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] graceful_shutdown_timeout = 60 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.594239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] host = np0000002381 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.594327 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] host_dvr_for_dhcp = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.594375 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] http_retries = 3 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.594457 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] instance_format = [instance: %(uuid)s] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.594534 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.594606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ipam_driver = internal {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.594674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ipv6_pd_enabled = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.594747 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] l3_ha = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.594833 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.594896 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_physical_name = {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.594938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_type = {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595014 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] log_color = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595093 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] log_config_append = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595165 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] log_dir = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595293 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] log_file = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] log_options = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595406 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval = 1 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595480 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval_type = days {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] log_rotation_type = none {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595608 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595654 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595696 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595780 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595780 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595901 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] max_allowed_address_pair = 10 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.595901 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] max_dns_nameservers = 5 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596004 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] max_header_line = 16384 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596004 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] max_l3_agents_per_router = 3 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_count = 30 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596099 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_size_mb = 200 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596165 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] max_routes = 30 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596252 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] max_subnet_host_routes = 20 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596322 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] my_ip = 10.4.3.231 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596387 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] my_ipv6 = ::1 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596482 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] network_auto_schedule = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596547 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] network_link_prefix = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_data_changes = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596712 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_status_changes = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596772 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] pagination_max_limit = -1 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596869 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] periodic_fuzzy_delay = 5 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] periodic_interval = 40 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.596980 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] publish_errors = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597024 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_burst = 0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597090 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_except_level = CRITICAL {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597159 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_interval = 0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597260 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] router_distributed = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597343 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] rpc_resources_processing_step = 20 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597406 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_max_timeout = 600 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597492 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_timeout = 60 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597560 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] rpc_state_report_workers = 0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597626 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] rpc_workers = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597715 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] send_events_interval = 2 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597842 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] service_plugins = ['ovn-router'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] setproctitle = on {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.597991 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] shell_completion = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] state_path = /opt/stack/data/neutron {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] syslog_log_facility = LOG_USER {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] tcp_keepidle = 600 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598305 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] transport_url = **** {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598383 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] use_journal = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598449 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] use_json = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598487 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] use_stderr = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598555 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] use_syslog = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598617 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] vlan_qinq = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598682 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] vlan_transparent = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] wsgi_default_pool_size = 100 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] wsgi_keep_alive = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598850 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598894 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] wsgi_server_debug = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 09 00:51:03.598985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.disable_process_locking = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599085 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599198 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler.connection_string = messaging:// {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599285 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler.enabled = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599355 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_doc_type = notification {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_size = 10000 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599486 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_time = 2m {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler.filter_error_trace = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599617 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler.hmac_keys = **** {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599676 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler.sentinel_service_name = mymaster {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler.socket_timeout = 0.1 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_requests = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_sqlalchemy = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.599969 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.process_tags = {} {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.service_name_prefix = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600114 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] profiler_otlp.service_name_prefix = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600213 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_process_name = {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600456 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600515 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600614 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_new_defaults = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600657 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_scope = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_default_rule = default {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600745 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600778 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600812 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600891 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.600891 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601015 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601015 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_timeout = 60.0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601287 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] service_providers.service_provider = [] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601351 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_section = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601351 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_type = password {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_url = https://10.4.3.231/identity {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.cafile = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601492 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.certfile = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601571 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.collect_timing = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_id = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_name = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_id = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_name = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.endpoint_type = public {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.insecure = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.keyfile = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.password = **** {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_id = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601972 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_name = Default {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.601972 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.project_id = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.project_name = service {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.region_name = RegionOne {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602109 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.split_loggers = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602109 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.system_scope = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.timeout = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.trust_id = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602243 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_id = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_name = Default {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602318 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.user_id = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602318 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] nova.username = nova {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602379 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_section = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_type = password {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602455 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] placement.cafile = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602488 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] placement.certfile = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] placement.collect_timing = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] placement.endpoint_type = public {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] placement.insecure = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602623 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] placement.keyfile = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602623 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] placement.region_name = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602686 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] placement.split_loggers = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602686 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] placement.timeout = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602749 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_section = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_type = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602816 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.cafile = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602849 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.certfile = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602881 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.collect_timing = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retries = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retry_delay = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.602985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.enable_notifications = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603075 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.endpoint_override = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603075 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.insecure = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.interface = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.keyfile = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603219 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.max_version = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603252 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.min_version = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603295 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.region_name = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603295 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.retriable_status_codes = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_name = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_type = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.split_loggers = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603456 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retries = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retry_delay = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.timeout = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.valid_interfaces = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603586 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ironic.version = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603586 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] cli_script.dry_run = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.603984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.hostname = np0000002381 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_compression = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.604990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.605520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.605520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.605520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.605520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.605520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_ca_file = {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.605520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_cert_file = {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.605520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.605520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_hostname_verification = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.605520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_key_file = {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_version = {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.use_queue_manager = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.driver = [] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.retry = -1 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.topics = ['notifications'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.transport_url = **** {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler_interval = 1 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.log_dir = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.default_quota = -1 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_floatingip = 50 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_network = 100 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_port = 500 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_rbac_policy = 10 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_router = 10 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.606544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group = 10 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group_rule = 100 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_subnet = 100 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.track_quota_usage = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep.group = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep.helper_command = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep.log_daemon_traceback = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep.logger_name = oslo_privsep.daemon {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep.thread_pool_size = 12 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep.user = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.capabilities = [21, 12] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.group = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.helper_command = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.log_daemon_traceback = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.thread_pool_size = 12 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.user = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.group = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.helper_command = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.thread_pool_size = 12 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.user = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.capabilities = [21] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.group = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.607914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.helper_command = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.log_daemon_traceback = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.thread_pool_size = 12 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.user = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.capabilities = [12] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.group = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.helper_command = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.log_daemon_traceback = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.thread_pool_size = 12 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.user = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.capabilities = [12, 21] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.group = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.helper_command = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.log_daemon_traceback = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.logger_name = oslo_privsep.daemon {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.thread_pool_size = 12 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.608774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.user = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609024 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609024 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609024 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] AGENT.use_helper_for_ns_read = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609024 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_ipset = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609024 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_security_group = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609024 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.firewall_driver = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609024 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.permitted_ethertypes = [] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609024 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] experimental.ipv6_pd_enabled = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2.external_network_type = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2.mechanism_drivers = ['ovn'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2.overlay_ip_version = 4 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2.path_mtu = 0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2.physical_network_mtus = {} {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2.project_network_types = ['geneve'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_flat.flat_networks = ['public'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.max_header_size = 38 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] network_log.burst_limit = 25 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] network_log.local_output_log_base = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] network_log.rate_limit = 100 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_connection = **** {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_slave_connection = **** {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.backend = sqlalchemy {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.connection = **** {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.connection_debug = 0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.connection_parameters = {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.connection_recycle_time = 3600 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.connection_trace = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.db_inc_retry_interval = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retries = 20 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.609784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retry_interval = 10 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.db_retry_interval = 1 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.max_overflow = 50 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.max_pool_size = 5 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.max_retries = 10 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_sql_mode = TRADITIONAL {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_wsrep_sync_wait = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.pool_timeout = None {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.retry_interval = 10 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.slave_connection = **** {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.sqlite_synchronous = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.synchronous_reader = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] database.use_db_reconnect = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_rx = 1000 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_tx = 100 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_mult = 3 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.broadcast_arps_to_all_routers = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.dhcp_default_lease_time = 43200 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_records_ovn_owned = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_servers = [] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.enable_distributed_floating_ip = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.fdb_age_threshold = 300 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ha_failover_strategy = normal {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.live_migration_activation_strategy = rarp {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.localnet_learn_fdb = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.mac_binding_age_threshold = 300 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.neutron_sync_mode = log {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp4_global_options = {} {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp6_global_options = {} {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_l3_scheduler = leastloaded {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_metadata_enabled = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_connection = ['ssl:10.4.3.231:6641'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.610818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_router_indirect_snat = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_connection = ['ssl:10.4.3.231:6642'] {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovs_create_tap = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_connection_timeout = 180 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_log_level = INFO {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_probe_interval = 60000 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_retry_max_interval = 180 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.stateless_nat_enabled = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] OVS.bridge_mac_table_size = 50000 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_reports = True {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_unregistered = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.611703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_snooping_enable = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.612177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] OVS.ovsdb_timeout = 10 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.612177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.fdb_removal_limit = 0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.612177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.ignore_lsp_down = False {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.612177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 09 00:51:03.612177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=89255) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3263}} Jun 09 00:51:03.612177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG cotyledon._service_worker [-] Run service ServiceWrapper(0) [89255] {{(pid=89255) wait_forever /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/_service_worker.py:218}} Jun 09 00:51:03.612565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-496796', 'neutron.services.pvlan.drivers.ovn.driver._initialize_pvlan_pg_drop-8196243791152', 'neutron.services.ovn_l3.plugin.OVNL3RouterPlugin._post_fork_initialize-12519'] for process (None), after_init {{(pid=89255) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 09 00:51:03.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbNbOvnIdl for MaintenanceWorker with retry Jun 09 00:51:03.654025 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Switch.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 09 00:51:03.654235 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Router.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 09 00:51:03.654235 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Load_Balancer.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 09 00:51:03.654392 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Copp.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.654392 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Meter.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.654513 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Load_Balancer_Group.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.654584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index HA_Chassis_Group.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.654652 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Address_Set.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.654879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Connection.target {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.654879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Gateway_Chassis.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.654879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Mirror.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.654879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Template_Var.chassis {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.655149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Router_Port.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.655149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Port_Group.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.655149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Switch_Port.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.658321 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.231:6641: connecting... Jun 09 00:51:03.663275 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.231:6641: connected Jun 09 00:51:03.685906 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbSbOvnIdl for MaintenanceWorker with retry Jun 09 00:51:03.705435 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Chassis.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 09 00:51:03.708227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Port_Binding.logical_port {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 09 00:51:03.708227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Private.name {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.708227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Datapath_Binding.tunnel_key {{(pid=89255) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 09 00:51:03.708464 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.231:6642: connecting... Jun 09 00:51:03.713871 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.231:6642: connected Jun 09 00:51:03.745365 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_update 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.745509 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.745549 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_create 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.745640 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_update 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.745707 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.745772 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port precommit_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.745981 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.portforwarding.drivers.ovn.driver.OVNPortForwarding.register-12542'] for port_forwarding_plugin (None), after_init {{(pid=89255) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 09 00:51:03.746039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_create 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.746109 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_create 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.746243 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_update 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.746291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.746357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.746505 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension Jun 09 00:51:03.746589 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.placement [-] Starting OVNClientPlacementExtension Jun 09 00:51:03.746951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment before_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.747051 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete_associations 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.747339 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.747424 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.747490 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_create 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.747558 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.747644 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.747716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.747775 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.747832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.747891 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_update 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.819432 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.819533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.819576 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_create 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.819633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_update 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.821948 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group after_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.822036 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.822155 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.logapi.drivers.ovn.driver.DriverBase._register-1336986'] for logging-plugin (None), after_init {{(pid=89255) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 09 00:51:03.822320 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_create 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.822388 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_update 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:03.822458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=89255) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 09 00:51:13.320040 np0000002381 neutron-ovn-maintenance-worker[88492]: DEBUG dbcounter [-] [88492] Writing DB stats neutron:SELECT=4 {{(pid=88492) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 00:51:13.823135 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Northbound DB sync process started @ 2026-06-09 00:51:13.822906 {{(pid=89255) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:99}} Jun 09 00:51:13.824560 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync port groups started @ 2026-06-09 00:51:13.824320 {{(pid=89255) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:132}} Jun 09 00:51:13.873781 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writer thread running {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} Jun 09 00:51:13.884174 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync port groups completed @ 2026-06-09 00:51:13.883942 {{(pid=89255) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:196}} Jun 09 00:51:13.884354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync networks, ports and DHCP options started @ 2026-06-09 00:51:13.884163 {{(pid=89255) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1152}} Jun 09 00:51:14.191603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.services.segments.db [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] neutron.services.segments.plugin.Plugin method get_segments called with arguments (,) {'filters': {'network_id': ['dc15bda6-1b51-4a9e-af84-ac570d199c11'], 'is_dynamic': False}} {{(pid=89255) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Jun 09 00:51:14.197782 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.services.segments.db [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] neutron.services.segments.plugin.Plugin method get_segments called with arguments (,) {'filters': {'network_id': ['b3e28628-ad68-4ca3-b181-b39c995a69e0'], 'is_dynamic': False}} {{(pid=89255) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Jun 09 00:51:14.203287 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync metadata ports started {{(pid=89255) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1097}} Jun 09 00:51:14.272435 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync metadata ports completed {{(pid=89255) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1149}} Jun 09 00:51:14.272597 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync DHCP options for Neutron subnets started {{(pid=89255) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:949}} Jun 09 00:51:14.327149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync DHCP options for Neutron subnets completed @ 2026-06-09 00:51:14.326948 {{(pid=89255) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1022}} Jun 09 00:51:14.327327 np0000002381 neutron-ovn-maintenance-worker[89255]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] Port found in OVN NB DB but not in Neutron, port_id=8596f435-902b-41b6-ae71-abd9b91ab35d Jun 09 00:51:14.327611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 09 00:51:14.327795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned started {{(pid=89255) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1027}} Jun 09 00:51:14.327880 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned completed {{(pid=89255) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1086}} Jun 09 00:51:14.327960 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync networks, ports and DHCP options completed @ 2026-06-09 00:51:14.327873 {{(pid=89255) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1321}} Jun 09 00:51:14.328069 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync ACLs started @ 2026-06-09 00:51:14.327980 {{(pid=89255) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:239}} Jun 09 00:51:14.361388 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync ACLs completed @ 2026-06-09 00:51:14.361131 {{(pid=89255) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:431}} Jun 09 00:51:14.361555 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync routers and router ports started @ 2026-06-09 00:51:14.361390 {{(pid=89255) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:736}} Jun 09 00:51:14.506109 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 09 00:51:14.506343 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync routers and router ports completed @ 2026-06-09 00:51:14.506179 {{(pid=89255) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:944}} Jun 09 00:51:14.506497 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync port QoS policies started @ 2026-06-09 00:51:14.506347 {{(pid=89255) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1369}} Jun 09 00:51:14.506565 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] Starting OVNClientQosExtension Jun 09 00:51:14.521244 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync port QoS policies completed @ 2026-06-09 00:51:14.521018 {{(pid=89255) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1392}} Jun 09 00:51:14.521444 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync Floating IP QoS policies started @ 2026-06-09 00:51:14.521227 {{(pid=89255) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1397}} Jun 09 00:51:14.521918 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] Starting OVNClientQosExtension Jun 09 00:51:14.543706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync Floating IP QoS policies completed @ 2026-06-09 00:51:14.543383 {{(pid=89255) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1420}} Jun 09 00:51:14.543974 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync Floating IP NAT rules started @ 2026-06-09 00:51:14.543702 {{(pid=89255) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1425}} Jun 09 00:51:14.544091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-NB Sync Floating IP NAT rules completed @ 2026-06-09 00:51:14.543939 {{(pid=89255) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1451}} Jun 09 00:51:14.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-8a98b6de-0dd8-4a70-9bc8-fd88a671f140 None None] OVN-Northbound DB sync process completed @ 2026-06-09 00:51:14.544038 {{(pid=89255) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:112}} Jun 09 00:51:23.824763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Southbound DB sync process started @ 2026-06-09 00:51:23.824516 {{(pid=89255) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1480}} Jun 09 00:51:23.825311 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension Jun 09 00:51:23.826456 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-54405037-e2b3-4c1c-91a0-7602e40aae28 None None] OVN-SB Sync hostname and physical networks started @ 2026-06-09 00:51:23.825933 {{(pid=89255) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1511}} Jun 09 00:51:23.829427 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.829623 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.830283 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_dhcp_options_consistency {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.830359 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fair_meter_consistency {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.830603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fdb_aging_settings {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.830603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_igmp_snoop_support {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.830603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_inconsistencies {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.830603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_global_dhcp_opts {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.830721 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.830721 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.830784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_provider_distributed_ports {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.830784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.830841 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.830883 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.832613 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.configure_nb_global {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.832697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.832769 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_fip_distributed_flag {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.832769 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_network_type_and_physnet {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.833021 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_ovn_owned_dns_option {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.833123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_ha_failover {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.833259 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.834029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-54405037-e2b3-4c1c-91a0-7602e40aae28 None None] Agents populated in the agent cache: {{(pid=89255) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:306}} Jun 09 00:51:23.834179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_mac_aging_settings {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.834323 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-54405037-e2b3-4c1c-91a0-7602e40aae28 None None] - Agent type: OVN Controller Gateway agent, hostname: np0000002381, name: 57c730f8-2d1f-4abc-ba9f-0689d3ced45d {{(pid=89255) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:308}} Jun 09 00:51:23.834467 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_distributed_flag {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.834595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-54405037-e2b3-4c1c-91a0-7602e40aae28 None None] - Agent type: OVN Neutron agent, hostname: np0000002381, name: 57c730f8-2d1f-4abc-ba9f-0689d3ced45d {{(pid=89255) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:308}} Jun 09 00:51:23.835433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_security_group_with_address_group {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.835580 np0000002381 neutron-ovn-maintenance-worker[89255]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-54405037-e2b3-4c1c-91a0-7602e40aae28 None None] New host np0000002381 found in OVN SB DB, but not in Neutron. Add its SegmentHostMapping in Neutron Jun 09 00:51:23.835812 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_virtual_port_parent_hostname {{(pid=89255) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 09 00:51:23.836685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.836970 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] Maintenance task thread has started Jun 09 00:51:23.837052 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] MaintenanceWorker process has finished the post initialization Jun 09 00:51:23.837548 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): ovsdb-client transact ssl:10.4.3.231:6641 --timeout 180 -p /opt/stack/data/CA/int-ca/private/devstack-cert.key -c /opt/stack/data/CA/int-ca/devstack-cert.crt -C /opt/stack/data/CA/int-ca/ca-chain.pem ["OVN_Northbound", {"op": "wait", "timeout": 0, "table": "Port_Group", "where": [["name", "==", "pvlan_pg_drop"]], "until": "==", "rows": []}, {"op": "insert", "table": "ACL", "row": {"action": "drop", "direction": "to-lport", "match": "outport == @pvlan_pg_drop && ip", "priority": 1010}, "uuid-name": "pvlandroptoport"}, {"op": "insert", "table": "ACL", "row": {"action": "drop", "direction": "from-lport", "match": "inport == @pvlan_pg_drop && ip", "priority": 1010}, "uuid-name": "pvlandropfromport"}, {"op": "insert", "table": "Port_Group", "row": {"name": "pvlan_pg_drop", "acls": ["set", [["named-uuid", "pvlandroptoport"], ["named-uuid", "pvlandropfromport"]]]}}] {{(pid=89255) execute /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:440}} Jun 09 00:51:23.838095 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.838208 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.838292 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.838383 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.838452 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.838769 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.838837 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.838912 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.838971 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.839580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.839580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.839580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.839580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.839580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.839580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.839805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.839805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.839805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.839805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.839805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.840172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.840172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.840172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.840172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' {{(pid=89255) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 09 00:51:23.863793 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG oslo_concurrency.lockutils [None req-54405037-e2b3-4c1c-91a0-7602e40aae28 None None] Acquiring lock "update_segment_host_mapping" by "neutron.services.segments.db.update_segment_host_mapping" {{(pid=89255) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} Jun 09 00:51:23.864027 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG oslo_concurrency.lockutils [None req-54405037-e2b3-4c1c-91a0-7602e40aae28 None None] Lock "update_segment_host_mapping" acquired by "neutron.services.segments.db.update_segment_host_mapping" :: waited 0.001s {{(pid=89255) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} Jun 09 00:51:23.873736 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG oslo_concurrency.processutils [-] CMD "ovsdb-client transact ssl:10.4.3.231:6641 --timeout 180 -p /opt/stack/data/CA/int-ca/private/devstack-cert.key -c /opt/stack/data/CA/int-ca/devstack-cert.crt -C /opt/stack/data/CA/int-ca/ca-chain.pem ["OVN_Northbound", {"op": "wait", "timeout": 0, "table": "Port_Group", "where": [["name", "==", "pvlan_pg_drop"]], "until": "==", "rows": []}, {"op": "insert", "table": "ACL", "row": {"action": "drop", "direction": "to-lport", "match": "outport == @pvlan_pg_drop && ip", "priority": 1010}, "uuid-name": "pvlandroptoport"}, {"op": "insert", "table": "ACL", "row": {"action": "drop", "direction": "from-lport", "match": "inport == @pvlan_pg_drop && ip", "priority": 1010}, "uuid-name": "pvlandropfromport"}, {"op": "insert", "table": "Port_Group", "row": {"name": "pvlan_pg_drop", "acls": ["set", [["named-uuid", "pvlandroptoport"], ["named-uuid", "pvlandropfromport"]]]}}]" returned: 0 in 0.036s {{(pid=89255) execute /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:468}} Jun 09 00:51:23.882481 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron_lib.callbacks.manager [None req-54405037-e2b3-4c1c-91a0-7602e40aae28 None None] Publish callbacks ['neutron.plugins.ml2.plugin.DhcpAgentSchedulerDbMixin.auto_schedule_new_network_segments-401945', 'neutron.services.segments.plugin.NovaSegmentNotifier._notify_host_addition_to_aggregate-2702772'] for segment_host_mapping (None), after_create {{(pid=89255) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 09 00:51:23.891683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.services.segments.db [None req-54405037-e2b3-4c1c-91a0-7602e40aae28 None None] Segments {'6a91ff68-89ba-411b-b54f-ff58910954a3'} mapped to the host np0000002381 {{(pid=89255) update_segment_host_mapping /opt/stack/neutron/neutron/services/segments/db.py:249}} Jun 09 00:51:23.894823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG oslo_concurrency.lockutils [None req-54405037-e2b3-4c1c-91a0-7602e40aae28 None None] Lock "update_segment_host_mapping" "released" by "neutron.services.segments.db.update_segment_host_mapping" :: held 0.031s {{(pid=89255) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} Jun 09 00:51:23.894995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-54405037-e2b3-4c1c-91a0-7602e40aae28 None None] OVN-SB Sync hostname and physical networks completed @ 2026-06-09 00:51:23.894840 {{(pid=89255) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1538}} Jun 09 00:51:23.914763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:23.915057 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_baremetal_ports_dhcp_options {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:23.916085 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Sync baremetal port DHCP options with configuration. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:23.989676 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_baremetal_ports_dhcp_options finished in 0.073 seconds Jun 09 00:51:23.989876 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:23.990056 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_dhcp_options_consistency {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:23.990169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Backfill network_id on DHCP_Options external_ids. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.037211 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_dhcp_options_consistency finished in 0.045 seconds Jun 09 00:51:24.037211 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.037211 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fair_meter_consistency {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.037211 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Refresh ACL logging fair meter after configuration reload. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.037211 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fair_meter_consistency finished in 0.000 seconds Jun 09 00:51:24.037211 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.037563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.037563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.045624 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:24.112477 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:24.113863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.118654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:24.118654 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.118654 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:24.118654 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:24.118654 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.118654 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:24.118654 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.118654 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:24.123675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.082 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.123675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:24.123675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:24.123675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:24.123675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.123675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:24.123675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:24.123675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.123675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:24.123675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.123675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.123675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.124153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:24.124581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.124581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:24.126993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:24.127653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.128349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:24.128349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.128349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:24.128349 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_for_igmp_snoop_support {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.128349 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Check for IGMP snoop support. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.128349 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_for_igmp_snoop_support finished in 0.000 seconds Jun 09 00:51:24.128349 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.128349 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.128694 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.349416 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: No inconsistencies found. Skipping {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:437}} Jun 09 00:51:24.349416 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_for_inconsistencies finished in 0.221 seconds Jun 09 00:51:24.349655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_global_dhcp_opts {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.349655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Check global DHCP options consistency. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.349799 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_global_dhcp_opts finished in 0.000 seconds Jun 09 00:51:24.350052 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.350151 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_localnet_port_has_learn_fdb {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.350246 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Ensure localnet ports have learn-fdb set per configuration. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.351371 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 09 00:51:24.352629 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_localnet_port_has_learn_fdb finished in 0.002 seconds Jun 09 00:51:24.352770 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.352871 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_network_broadcast_arps_to_all_routers {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.352942 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Sync broadcast-arps-to-all-routers on external networks. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.407213 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_network_broadcast_arps_to_all_routers finished in 0.053 seconds Jun 09 00:51:24.407213 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.407213 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_provider_distributed_ports {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.407213 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Align reside-on-redirect-chassis on VLAN/FLAT distributed ports. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.482250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 09 00:51:24.482493 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-54405037-e2b3-4c1c-91a0-7602e40aae28 None None] OVN-Southbound DB sync process completed @ 2026-06-09 00:51:24.482337 {{(pid=89255) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1488}} Jun 09 00:51:24.483130 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_provider_distributed_ports finished in 0.077 seconds Jun 09 00:51:24.483318 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.483517 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_redirect_type_router_gateway_ports {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.483630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Align router gateway redirect-type for provider VLAN/FLAT networks. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.530171 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_redirect_type_router_gateway_ports finished in 0.046 seconds Jun 09 00:51:24.530328 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.530470 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.530470 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.531452 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:24.531674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.532132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:24.532132 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.532132 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:24.532132 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:24.532132 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.532132 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:24.532132 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.532132 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:24.533000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.533000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:24.533000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:24.533000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:24.533000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.533000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:24.533000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:24.533000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.533000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:24.533000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.533000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.533000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.533289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:24.533522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.533522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:24.533747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:24.534040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.534403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:24.534403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.534403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:24.534403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: cleanup_old_hash_ring_nodes {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.534403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Purge stale hash ring nodes older than five days. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.538276 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.db.ovn_hash_ring_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Cleaned up Hash Ring nodes older than 5 days Jun 09 00:51:24.538450 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task cleanup_old_hash_ring_nodes finished in 0.004 seconds Jun 09 00:51:24.538716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: configure_nb_global {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.538841 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply ovn_nb_global settings to NB_Global options. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.539113 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:24.539688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:24.539984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.540681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:24.540681 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.540681 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:24.540681 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:24.540681 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.540681 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:24.540681 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.540681 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:24.541939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:24.542314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.542314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:24.542314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.542314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:24.542314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:24.542314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.542314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:24.542314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.542314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:24.544427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task configure_nb_global failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.544427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:24.544427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:24.544427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:24.544427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.544427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global Jun 09 00:51:24.544427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:24.544427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:24.544427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:24.544427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.544427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:24.544713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:24.544966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.544966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:24.544966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.544966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' (it runs every 86400.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:24.545256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.545607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:24.545928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:24.545928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.545928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:24.545928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.545928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:24.545928 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 09 00:51:24.546049 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.546225 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.546275 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.546656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:51:24.546926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:24.547108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.547625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:24.547625 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.547625 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:24.547625 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:24.547625 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.547625 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:24.547625 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.547625 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:24.548443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:24.548743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.548743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:24.548743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.548743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:24.548743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:24.548743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.548743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:24.548743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.548743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:24.548994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.548994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:24.548994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:24.548994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:24.548994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.548994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:24.548994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:24.548994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:51:24.548994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:24.548994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:24.548994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.548994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:24.549263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:24.549540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.549540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:24.549540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.549540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:24.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.550202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:24.550536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.550536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:24.550536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:24.550536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.550536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:24.550536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.550536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:24.550536 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_network_type_and_physnet {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.550536 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set network type and physnet external_ids on logical switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.556217 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_network_type_and_physnet finished in 0.006 seconds Jun 09 00:51:24.556371 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.556483 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_ovn_owned_dns_option {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.556591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply ovn_owned option on DNS records per configuration. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.556778 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_ovn_owned_dns_option finished in 0.000 seconds Jun 09 00:51:24.556868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.556964 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.557046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.557200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:24.557668 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:24.557872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.558288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:24.558288 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.558288 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:24.558288 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:24.558288 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.558288 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:24.558288 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.558288 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:24.558856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:24.559162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.559162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:24.559162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.559162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:24.559162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:24.559162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.559162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:24.559162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.559162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:24.559577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.559577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:24.559577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:24.559577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:24.559577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.559577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:24.559577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:24.559577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:24.559577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:24.559577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.559577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:24.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:24.560093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.560093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:24.560093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.560093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.560378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:24.560735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:24.561059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.561059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:24.561059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.561059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:24.561059 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_lrouter_ports_ext_ids_name_prefix {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.561059 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: ('Check missing prefix router_name', 'in external_ids of LRPs.') {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.561324 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_lrouter_ports_ext_ids_name_prefix finished in 0.000 seconds Jun 09 00:51:24.561425 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.561553 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.561628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.561738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:24.562119 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:24.562327 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:24.562436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.562799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:24.562799 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.562799 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:24.562799 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:24.562799 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.562799 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:24.562799 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.562799 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:24.563364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.563364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:24.563364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:24.563364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:24.563364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.563364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:24.563364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:24.563364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.563364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:24.563364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.563364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.563364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.563621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:24.563858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.563858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:24.563994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:24.564658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:24.565399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:24.565399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:24.565399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:24.565399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_router_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.565399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Sync router.distributed with distributed floating IP config. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_router_distributed_flag finished in 0.003 seconds Jun 09 00:51:24.567521 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.567641 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_security_group_with_address_group {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:24.567721 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Repair Address_Sets and ACLs for address group rules. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:24.576747 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_security_group_with_address_group finished in 0.009 seconds Jun 09 00:51:24.576863 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:24.577784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89255) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 09 00:51:29.036132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:29.038032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:29.038032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:29.038032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:29.064922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:29.065086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:29.066779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:29.067153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.067439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:29.067703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:29.068061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:29.068061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:29.068061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.068061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:29.068061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:29.068061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.068061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:29.068061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.068061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:29.530679 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:29.531494 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:29.531494 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:29.531778 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:29.531889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:29.534578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:29.535149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.535598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:29.536081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:29.536722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:29.536722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:29.536722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:29.536722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.536722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:29.536722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:29.536722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.536722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:29.536722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.536722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:29.546285 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:29.546934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:29.546934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:29.546934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:51:29.547200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:29.547398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.550579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:29.551153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:29.551583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.551961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:29.552354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:29.552766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:29.553281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:29.553281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:29.553281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.553281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:29.553281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:29.553281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.553281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:29.553281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.553281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:29.557169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:29.557826 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:29.557826 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:29.557826 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:29.558229 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:29.558451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.561700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:29.562280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:29.562729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.563098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:29.563451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:29.564003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:29.564528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:29.564528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.564528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:29.564528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:29.564528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.564528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:29.564528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.564528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:29.564528 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:29.565054 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:29.565456 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:29.565728 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:29.566206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:29.566485 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:29.566631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:29.569123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.569990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:29.570664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:29.571138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:29.571138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:29.571138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:29.571138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:29.571138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:29.571138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:29.571138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:29.571138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:29.571138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:29.571138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:34.036994 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:34.037601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:34.037601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:34.038053 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:34.066947 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:34.067085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.067595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:34.067595 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.067595 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:34.067595 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:34.067595 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.067595 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:34.067595 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.067595 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:34.068355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.068355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:34.068355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:34.068355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:34.068355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.068355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:34.068355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:34.068355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.068355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:34.068355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:34.068355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:34.068355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.068703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:34.068999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.068999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:34.069206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:34.070216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.070780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:34.070780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.070780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:34.531255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:34.531935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:34.532042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:34.532644 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:34.532821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.533279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:34.533279 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.533279 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:34.533279 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:34.533279 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.533279 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:34.533279 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.533279 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:34.534089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.534089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:34.534089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:34.534089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:34.534089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.534089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:34.534089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:34.534089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.534089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:34.534089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:34.534089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:34.534089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.534474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:34.534831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.534831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:34.534899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:34.535355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.535907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:34.535907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.535907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:34.546882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:34.547269 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:34.547398 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:34.547542 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:51:34.547961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:34.548157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.548603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:34.548603 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.548603 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:34.548603 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:34.548603 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.548603 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:34.548603 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.548603 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:34.549250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:34.549720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:34.549720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:34.549720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.549720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:34.549720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:34.549720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.549720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:34.549720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.549720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:34.549949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.549949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:34.549949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:34.549949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:34.549949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.549949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:34.549949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:34.549949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:51:34.549949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:34.549949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:34.549949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.549949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:34.550302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:34.550770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:34.551158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:34.551681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:34.551681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:34.551681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:34.551681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:34.551681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:34.551681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.551681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:34.551681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:34.551681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.551681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:34.551681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.551681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:34.557735 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:34.558125 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:34.558255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:34.558378 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:34.558708 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:34.558893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.559312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:34.559312 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.559312 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:34.559312 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:34.559312 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.559312 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:34.559312 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.559312 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:34.559948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:34.560413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:34.560413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:34.560413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.560413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:34.560413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:34.560413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.560413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:34.560413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.560413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:34.560664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.560664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:34.560664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:34.560664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:34.560664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.560664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:34.560664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:34.560664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:34.560664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:34.560664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.560664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:34.560959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:34.561327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:34.561750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:34.562232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:34.562232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:34.562232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:34.562232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:34.562232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.562232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:34.562232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:34.562232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.562232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:34.562232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.562232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:34.565245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:34.565582 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:34.565697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:34.565839 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:34.566156 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:34.566357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:34.566422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.566824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:34.566824 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.566824 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:34.566824 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:34.566824 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.566824 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:34.566824 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.566824 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:34.567535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.567535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:34.567535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:34.567535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:34.567535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.567535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:34.567535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:34.567535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.567535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:34.567535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:34.567535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:34.567535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.567903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:34.568276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.568276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:34.568734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:34.569357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:34.570234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:34.570234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:34.570234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:39.037422 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:39.038029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:39.038097 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:39.039122 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:39.074613 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:39.074665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.075218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:39.075218 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.075218 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:39.075218 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:39.075218 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.075218 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:39.075218 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.075218 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:39.076049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.076049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:39.076049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:39.076049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:39.076049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.076049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:39.076049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:39.076049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.076049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:39.076049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:39.076049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:39.076049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.076620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:39.077061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.077061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:39.077248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:39.077682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.078044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:39.078044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.078044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:39.531977 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:39.532409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:39.532551 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:39.533108 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:39.533320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.533900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:39.533900 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.533900 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:39.533900 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:39.533900 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.533900 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:39.533900 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.533900 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:39.534847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.534847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:39.534847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:39.534847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:39.534847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.534847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:39.534847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:39.534847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.534847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:39.534847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:39.534847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:39.534847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.535096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:39.535386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.535386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:39.535929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:39.536271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.536626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:39.536626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.536626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:39.547344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:39.547822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:39.547951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:39.548080 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:51:39.548513 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:39.548758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.549343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:39.549343 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.549343 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:39.549343 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:39.549343 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.549343 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:39.549343 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.549343 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:39.550108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:39.550623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:39.550623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:39.550623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.550623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:39.550623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:39.550623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.550623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:39.550623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.550623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:39.552786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.552786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:39.552786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:39.552786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:39.552786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.552786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:39.552786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:39.552786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:51:39.552786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:39.552786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:39.552786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.552786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:39.553155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:39.553456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:39.553768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:39.554141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:39.554141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:39.554141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:39.554141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:39.554141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:39.554141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.554141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:39.554141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:39.554141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.554141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:39.554141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.554141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:39.558745 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:39.558745 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:39.558745 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:39.559096 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:39.559504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:39.559800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.560284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:39.560284 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.560284 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:39.560284 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:39.560284 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.560284 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:39.560284 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.560284 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:39.560840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:39.561317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:39.561317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:39.561317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.561317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:39.561317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:39.561317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.561317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:39.561317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.561317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:39.562036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.562036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:39.562036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:39.562036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:39.562036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.562036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:39.562036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:39.562036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:39.562036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:39.562036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.562036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:39.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:39.562834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:39.563276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:39.563826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:39.563826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:39.563826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:39.563826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:39.563826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.563826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:39.563826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:39.563826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.563826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:39.563826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.563826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:39.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:39.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:39.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:39.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:39.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:39.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:39.568848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:39.569240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:39.569639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:39.570173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:39.570173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:39.570173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:39.570173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:39.570173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:39.570173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:39.570173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:39.570173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:39.570173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:39.570173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:39.570173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:39.570173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:44.039618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:44.039618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:44.039618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:44.040380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:44.083274 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:44.083274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.083746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:44.083746 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.083746 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:44.083746 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:44.083746 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.083746 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:44.083746 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.083746 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:44.084790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.084790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:44.084790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:44.084790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:44.084790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.084790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:44.084790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:44.084790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.084790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:44.084790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:44.084790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:44.084790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.085162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:44.085603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.085603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:44.086176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:44.086641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.087153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:44.087153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.087153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:44.532512 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:44.533502 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:44.533618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:44.534850 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:44.534890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.535767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:44.535767 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.535767 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:44.535767 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:44.535767 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.535767 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:44.535767 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.535767 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:44.536618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.536618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:44.536618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:44.536618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:44.536618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.536618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:44.536618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:44.536618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.536618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:44.536618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:44.536618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:44.536618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.536865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:44.537120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.537120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:44.539332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:44.539827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.540404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:44.540404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.540404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:44.547842 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:44.548766 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:44.548952 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:44.549106 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:51:44.549509 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:44.549731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.552063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:44.552063 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.552063 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:44.552063 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:44.552063 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.552063 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:44.552063 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.552063 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:44.552858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:44.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:44.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:44.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:44.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:44.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:44.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:44.553641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.553641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:44.553641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:44.553641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:44.553641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.553641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:44.553641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:44.553641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:51:44.553641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:44.553641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:44.553641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.553641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:44.553905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:44.554157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.554157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:44.554157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.554157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:44.554395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:44.554657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:44.554971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.554971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:44.554971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:44.554971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.554971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:44.554971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.554971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:44.558573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:44.558841 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:44.558925 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:44.559048 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:44.559439 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:44.559757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.560348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:44.560348 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.560348 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:44.560348 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:44.560348 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.560348 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:44.560348 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.560348 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:44.561447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:44.561936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:44.561936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:44.561936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.561936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:44.561936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:44.561936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.561936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:44.561936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.561936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:44.562251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.562251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:44.562251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:44.562251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:44.562251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.562251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:44.562251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:44.562251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:44.562251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:44.562251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.562251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:44.562541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:44.563027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.563027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:44.563027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.563027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.563325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:44.563675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:44.563991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.563991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:44.563991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.563991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:44.566073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:44.567406 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:44.567542 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:44.567734 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:44.568204 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:44.568453 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:44.568541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.571605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:44.571605 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.571605 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:44.571605 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:44.571605 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.571605 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:44.571605 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.571605 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:44.573707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.573707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:44.573707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:44.573707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:44.573707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.573707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:44.573707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:44.573707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.573707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:44.573707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:44.573707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:44.573707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.574105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:44.574522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.574522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:44.574841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:44.575220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:44.575739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:44.575739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:44.575739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:49.039019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:49.041554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:49.041671 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:49.042246 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:49.077909 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:49.078114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:49.080370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:49.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.080947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:49.081205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:49.081537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:49.081537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:49.081537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.081537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:49.081537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:49.081537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.081537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:49.081537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.081537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:49.533612 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:49.534655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:49.534655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:49.534805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:49.534938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.535357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:49.535357 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.535357 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:49.535357 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:49.535357 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.535357 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:49.535357 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.535357 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:49.537913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.537913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:49.537913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:49.537913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:49.537913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.537913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:49.537913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:49.537913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.537913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:49.537913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:49.537913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:49.537913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.538310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:49.538631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:49.539002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.539465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:49.539465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:49.539465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.539465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:49.539465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.539465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:49.548825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:49.549658 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:49.549658 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:49.549658 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:51:49.549867 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:49.549941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.552216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:49.552620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:49.552854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.553072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:49.553295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:49.553598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:49.553978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:49.553978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:49.553978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.553978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:49.553978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:49.553978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.553978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:49.553978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.553978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:49.558973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:49.559656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:49.559656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:49.559656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:49.559815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:49.560007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.562893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:49.563911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:49.564786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.565259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:49.565638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:49.566042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:49.566590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:49.566590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.566590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:49.566590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:49.566590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.566590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:49.566590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.566590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:49.567459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:49.567846 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:49.568141 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:49.568467 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:49.568899 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:49.569103 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:49.569205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:49.571766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:49.572688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.573437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:49.574125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:49.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:49.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:49.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:49.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:49.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:49.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:49.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:49.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:49.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:49.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:54.041686 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:54.043882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:54.043882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:54.043882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:54.067412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:54.067651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.068249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:54.068249 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.068249 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:54.068249 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:54.068249 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.068249 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:54.068249 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.068249 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:54.069971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.069971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:54.069971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:54.069971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:54.069971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.069971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:54.069971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:54.069971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.069971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:54.069971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:54.069971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:54.069971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.070389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:54.070765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:54.071216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.071774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:54.071774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:54.071774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.071774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:54.071774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.071774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:54.535031 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:54.535973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:54.535973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:54.536245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:54.536426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.537293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:54.537293 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.537293 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:54.537293 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:54.537293 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.537293 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:54.537293 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.537293 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:54.538134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.538134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:54.538134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:54.538134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:54.538134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.538134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:54.538134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:54.538134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.538134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:54.538134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:54.538134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:54.538134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.538455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:54.538681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.538681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:54.539442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:54.539742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.540068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:54.540068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.540068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:54.549437 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:54.549793 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:54.549959 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:54.550105 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:51:54.550582 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:54.550778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.551295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:54.551295 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.551295 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:54.551295 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:54.551295 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.551295 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:54.551295 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.551295 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:54.552204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:54.552622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:54.552622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:54.552622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.552622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:54.552622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:54.552622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.552622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:54.552622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.552622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:54.553118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.553118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:54.553118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:54.553118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:54.553118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.553118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:54.553118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:54.553118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:51:54.553118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:54.553118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:54.553118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.553118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:54.553377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:54.553613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.553613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:54.553613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.553613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:54.554289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:54.554572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:54.554846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.554846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:54.554846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:54.554846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.554846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:54.554846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.554846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:54.559449 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:54.559773 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:54.559922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:54.560079 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:54.560574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:54.560831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.561393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:54.561393 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.561393 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:54.561393 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:54.561393 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.561393 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:54.561393 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.561393 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:54.562296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:54.562592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:54.562592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:54.562592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.562592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:54.562592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:54.562592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.562592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:54.562592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.562592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:54.563223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.563223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:54.563223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:54.563223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:54.563223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.563223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:54.563223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:54.563223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:54.563223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:54.563223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.563223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:54.563447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:54.563717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.563717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:54.563717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.563717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.564345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:54.564675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:54.564991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.564991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:54.564991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.564991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:54.568023 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:54.568369 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:54.568505 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:54.568743 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:54.569144 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:54.569401 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:54.569515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.570093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:54.570093 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.570093 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:54.570093 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:54.570093 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.570093 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:54.570093 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.570093 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:54.571237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.571237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:54.571237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:54.571237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:54.571237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.571237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:54.571237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:54.571237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.571237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:54.571237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:54.571237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:54.571237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.571508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:54.571729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.571729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:54.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:54.572535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:54.572845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:54.572845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:54.572845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:59.042643 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:59.043336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:59.043336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:59.043759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:59.068706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:59.068706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.069086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:59.069086 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.069086 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:59.069086 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:59.069086 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.069086 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:59.069086 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.069086 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:59.069792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.069792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:59.069792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:59.069792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:59.069792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.069792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:59.069792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:59.069792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.069792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:59.069792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:59.069792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:59.069792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.070035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:59.070308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.070308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:59.070994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:59.071895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.072483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:59.072483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.072483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:59.535621 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:59.535893 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:59.536006 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:59.538564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:59.538711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.539133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:59.539133 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.539133 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:59.539133 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:59.539133 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.539133 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:59.539133 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.539133 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:59.540051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.540051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:59.540051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:59.540051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:59.540051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.540051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:59.540051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:59.540051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.540051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:59.540051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:59.540051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:59.540051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.540299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:59.540625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.540625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:59.541199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:59.541478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.541788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:59.541788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.541788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:59.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:59.550096 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:59.550203 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:59.550353 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:51:59.550687 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:59.550844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.551245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:59.551245 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.551245 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:59.551245 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:59.551245 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.551245 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:59.551245 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.551245 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:59.551990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:59.552440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:59.552440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:59.552440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.552440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:59.552440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:59.552440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.552440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:59.552440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.552440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:59.552904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.552904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:59.552904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:59.552904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:59.552904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.552904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:59.552904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:59.552904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:51:59.552904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:59.552904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:59.552904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.552904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:59.553263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:59.553659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.553659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:59.553659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.553659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:59.553853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:59.554286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:59.554767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.554767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:59.554767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:59.554767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.554767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:59.554767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.554767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:59.559969 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:59.560250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:59.560403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:59.560483 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:59.561031 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:59.561296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.561917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:59.561917 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.561917 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:59.561917 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:59.561917 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.561917 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:59.561917 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.561917 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:51:59.562868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:51:59.563167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:59.563167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:51:59.563167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.563167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:51:59.563167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:51:59.563167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.563167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:51:59.563167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.563167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:51:59.563850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.563850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:59.563850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:59.563850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:59.563850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.563850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:59.563850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:59.563850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:59.563850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:59.563850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.563850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:59.564096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:59.564389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.564389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:59.564389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.564389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.565025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:59.565367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:59.565769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.565769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:59.565769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.565769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:51:59.568421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:51:59.568670 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:51:59.568778 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:51:59.568972 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:51:59.570828 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:59.571074 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:51:59.571175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.571801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:51:59.571801 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.571801 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:51:59.571801 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:51:59.571801 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.571801 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:51:59.571801 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.571801 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:51:59.572504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.572504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:51:59.572504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:59.572504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:51:59.572504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.572504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:59.572504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:59.572504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.572504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:59.572504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:59.572504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:51:59.572504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.573003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:51:59.573452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:51:59.573972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:51:59.574663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:51:59.574663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:51:59.574663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:51:59.574663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:51:59.574663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:51:59.574663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:04.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:04.043626 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:04.043626 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:04.044004 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:04.071001 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:04.071276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.071925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:04.071925 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.071925 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:04.071925 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:04.071925 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.071925 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:04.071925 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.071925 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:04.072774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.072774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:04.072774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:04.072774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:04.072774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.072774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:04.072774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:04.072774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.072774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:04.072774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:04.072774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:04.072774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.073112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:04.073526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.073526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:04.073862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:04.074360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.074918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:04.074918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.074918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:04.536042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:04.536357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:04.536476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:04.537418 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:04.537605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:04.539222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:04.539780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.540272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:04.540696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:04.541356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:04.541356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:04.541356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:04.541356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.541356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:04.541356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:04.541356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.541356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:04.541356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.541356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:04.550239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:04.550550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:04.550692 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:04.550795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:52:04.552157 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:04.552379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.552878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:04.552878 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.552878 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:04.552878 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:04.552878 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.552878 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:04.552878 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.552878 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:04.553547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:04.553995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:04.553995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:04.553995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.553995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:04.553995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:04.553995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.553995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:04.553995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.553995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:04.554301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.554301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:04.554301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:04.554301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:04.554301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.554301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:04.554301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:04.554301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:52:04.554301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:04.554301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:04.554301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.554301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:04.554671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:04.555070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:04.555472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:04.556064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:04.556064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:04.556064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:04.556064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:04.556064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:04.556064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.556064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:04.556064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:04.556064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.556064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:04.556064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.556064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:04.560314 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:04.560556 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:04.560646 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:04.560748 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:04.561092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:04.561366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.561807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:04.561807 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.561807 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:04.561807 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:04.561807 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.561807 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:04.561807 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.561807 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:04.562488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:04.562934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:04.562934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:04.562934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.562934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:04.562934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:04.562934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.562934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:04.562934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.562934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:04.563215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.563215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:04.563215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:04.563215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:04.563215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.563215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:04.563215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:04.563215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:04.563215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:04.563215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.563215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:04.563573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:04.563985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:04.564396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:04.564958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:04.564958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:04.564958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:04.564958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:04.564958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.564958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:04.564958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:04.564958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.564958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:04.564958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.564958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:04.568767 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:04.568996 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:04.569081 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:04.569282 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:04.569624 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:04.569822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:04.569912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.570299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:04.570299 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.570299 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:04.570299 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:04.570299 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.570299 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:04.570299 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.570299 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:04.570974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.570974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:04.570974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:04.570974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:04.570974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.570974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:04.570974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:04.570974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.570974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:04.570974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:04.570974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:04.570974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.571348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:04.571682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:04.572113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:04.572704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:04.572704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:04.572704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:04.572704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:04.572704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:04.572704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:09.043376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:09.043969 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:09.043969 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:09.044302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:09.076212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:09.076512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.077117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:09.077117 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.077117 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:09.077117 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:09.077117 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.077117 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:09.077117 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.077117 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:09.078118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.078118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:09.078118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:09.078118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:09.078118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.078118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:09.078118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:09.078118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.078118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:09.078118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:09.078118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:09.078118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.078487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:09.078796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.078796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:09.079754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:09.080059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.080525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:09.080525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.080525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:09.536541 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:09.536770 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:09.536855 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:09.537366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:09.537552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.537956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:09.537956 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.537956 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:09.537956 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:09.537956 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.537956 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:09.537956 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.537956 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:09.538807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.538807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:09.538807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:09.538807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:09.538807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.538807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:09.538807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:09.538807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.538807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:09.538807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:09.538807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:09.538807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.539166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:09.539576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.539576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:09.539776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:09.540218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.540613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:09.540613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.540613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:09.550632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:09.550821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:09.550897 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:09.550984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:52:09.551374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:09.551609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.552019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:09.552019 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.552019 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:09.552019 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:09.552019 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.552019 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:09.552019 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.552019 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:09.552666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:09.552958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:09.552958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:09.552958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.552958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:09.552958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:09.552958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.552958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:09.552958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.552958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:09.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:09.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:09.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:09.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:09.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:09.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:52:09.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:09.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:09.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:09.553566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:09.553832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.553832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:09.553832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.553832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:09.554005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:09.554351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:09.554720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.554720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:09.554720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:09.554720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.554720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:09.554720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.554720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:09.560703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:09.560894 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:09.560990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:09.561089 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:09.561448 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:09.561752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.562201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:09.562201 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.562201 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:09.562201 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:09.562201 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.562201 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:09.562201 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.562201 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:09.562835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:09.563247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:09.563247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:09.563247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.563247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:09.563247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:09.563247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.563247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:09.563247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.563247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:09.563666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.563666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:09.563666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:09.563666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:09.563666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.563666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:09.563666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:09.563666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:09.563666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:09.563666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.563666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:09.563907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:09.564279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.564279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:09.564279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.564279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.564710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:09.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:09.565530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.565530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:09.565530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.565530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:09.569149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:09.569402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:09.569522 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:09.569647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:09.569970 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:09.570239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:09.570422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:09.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:09.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:09.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:09.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:09.571714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.571714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:09.571714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:09.571714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:09.571714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.571714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:09.571714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:09.571714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.571714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:09.571714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:09.571714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:09.571714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.572004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:09.572297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.572297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:09.572818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:09.573140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:09.573495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:09.573495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:09.573495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:14.043835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:14.044360 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:14.044360 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:14.045205 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:14.046414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=547,neutron:INSERT=1,neutron:UPDATE=2,neutron:DELETE=1 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 00:52:14.074932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:14.075534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.076007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:14.076007 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.076007 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:14.076007 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:14.076007 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.076007 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:14.076007 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.076007 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:14.077164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.077164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:14.077164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:14.077164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:14.077164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.077164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:14.077164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:14.077164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.077164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:14.077164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:14.077164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:14.077164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.077673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:14.078055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:14.078533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.079044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:14.079044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:14.079044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.079044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:14.079044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.079044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:14.536918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:14.537131 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:14.537253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:14.537665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:14.537803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.538162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:14.538162 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.538162 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:14.538162 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:14.538162 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.538162 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:14.538162 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.538162 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:14.538785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.538785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:14.538785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:14.538785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:14.538785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.538785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:14.538785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:14.538785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.538785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:14.538785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:14.538785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:14.538785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.539017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:14.539239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.539239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:14.539455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:14.539741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.540066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:14.540066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.540066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:14.550890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:14.551031 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:14.551108 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:14.551206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:52:14.551428 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:14.551581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.551923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:14.551923 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.551923 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:14.551923 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:14.551923 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.551923 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:14.551923 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.551923 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:14.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:14.552679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:14.552679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:14.552679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.552679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:14.552679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:14.552679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.552679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:14.552679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.552679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:14.553030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.553030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:14.553030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:14.553030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:14.553030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.553030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:14.553030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:14.553030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:52:14.553030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:14.553030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:14.553030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.553030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:14.553279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:14.553513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.553513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:14.553513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.553513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:14.553721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:14.553992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:14.554356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.554356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:14.554356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:14.554356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.554356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:14.554356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.554356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:14.560947 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:14.561101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:14.561178 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:14.561291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:14.561518 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:14.561683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.562020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:14.562020 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.562020 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:14.562020 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:14.562020 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.562020 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:14.562020 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.562020 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:14.562575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:14.562840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:14.562840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:14.562840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.562840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:14.562840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:14.562840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.562840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:14.562840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.562840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:14.563109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.563109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:14.563109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:14.563109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:14.563109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.563109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:14.563109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:14.563109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:14.563109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:14.563109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.563109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:14.563350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:14.563599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.563599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:14.563599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.563599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.563796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:14.564107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:14.564466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.564466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:14.564466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.564466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:14.569414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:14.569570 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:14.569646 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:14.569757 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:14.570181 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:14.570430 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:14.570534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.570975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:14.570975 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.570975 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:14.570975 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:14.570975 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.570975 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:14.570975 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.570975 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:14.571548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.571548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:14.571548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:14.571548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:14.571548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.571548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:14.571548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:14.571548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.571548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:14.571548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:14.571548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:14.571548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.571746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:14.571933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.571933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:14.572200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:14.572525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:14.572829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:14.572829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:14.572829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:19.044293 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:19.046166 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:19.046166 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:19.046166 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:19.067892 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:19.068112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:19.069990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:19.070323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.070551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:19.070787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:19.071105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:19.071105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:19.071105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.071105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:19.071105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:19.071105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.071105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:19.071105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.071105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:19.537364 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:19.538136 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:19.538136 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:19.538525 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:19.538700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.539181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:19.539181 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.539181 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:19.539181 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:19.539181 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.539181 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:19.539181 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.539181 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:19.541452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.541452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:19.541452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:19.541452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:19.541452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.541452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:19.541452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:19.541452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.541452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:19.541452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:19.541452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:19.541452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.542121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:19.543061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:19.544079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.544535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:19.544535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:19.544535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.544535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:19.544535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.544535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:19.551176 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:19.551410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:19.551522 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:19.551597 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:52:19.551968 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:19.552133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.552764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:19.552764 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.552764 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:19.552764 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:19.552764 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.552764 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:19.552764 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.552764 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:19.553171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:19.553593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:19.553593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:19.553593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.553593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:19.553593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:19.553593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.553593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:19.553593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.553593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:19.553815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.553815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:19.553815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:19.553815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:19.553815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.553815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:19.553815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:19.553815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:52:19.553815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:19.553815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:19.553815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.553815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:19.554124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:19.554456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.554456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:19.554456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.554456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:19.554645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:19.555455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:19.556645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.556645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:19.556645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:19.556645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.556645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:19.556645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.556645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:19.561212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:19.561422 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:19.561531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:19.561599 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:19.561988 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:19.562222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.562677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:19.562677 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.562677 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:19.562677 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:19.562677 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.562677 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:19.562677 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.562677 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:19.563233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:19.563605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:19.563605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:19.563605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.563605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:19.563605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:19.563605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.563605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:19.563605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.563605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:19.563997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.563997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:19.563997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:19.563997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:19.563997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.563997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:19.563997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:19.563997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:19.563997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:19.563997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.563997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:19.564625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:19.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:19.565532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:19.565974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:19.565974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:19.565974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:19.565974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:19.565974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.565974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:19.565974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:19.565974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.565974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:19.565974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.565974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:19.569646 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:19.569807 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:19.569864 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:19.569998 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:19.570353 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:19.570552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:19.570635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.571045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:19.571045 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.571045 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:19.571045 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:19.571045 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.571045 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:19.571045 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.571045 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:19.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:19.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:19.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:19.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:19.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:19.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:19.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:19.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:19.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.572000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:19.572309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.572309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:19.572368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:19.572753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:19.573208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:19.573208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:19.573208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:24.045051 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:24.045656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:24.045656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:24.045907 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:24.078219 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:24.078219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.078219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:24.078219 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.078219 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:24.078219 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:24.078219 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.078219 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:24.078219 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.078219 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:24.078749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.078749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:24.078749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:24.078749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:24.078749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.078749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:24.078749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:24.078749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.078749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:24.078749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:24.078749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:24.078749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.078984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:24.079175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.079175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:24.080234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:24.080717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.081033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:24.081033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.081033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:24.538025 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:24.538298 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:24.538391 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:24.539269 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:24.539269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.539867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:24.539867 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.539867 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:24.539867 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:24.539867 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.539867 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:24.539867 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.539867 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:24.540929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.540929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:24.540929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:24.540929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:24.540929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.540929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:24.540929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:24.540929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.540929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:24.540929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:24.540929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:24.540929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.541494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:24.542153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:24.542590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.542916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:24.542916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:24.542916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.542916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:24.542916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.542916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:24.551563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:24.551782 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:24.551879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:24.552008 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:52:24.552539 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:24.552711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.553351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:24.553351 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.553351 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:24.553351 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:24.553351 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.553351 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:24.553351 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.553351 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:24.553857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:24.554275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:24.554275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:24.554275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.554275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:24.554275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:24.554275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.554275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:24.554275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.554275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:24.554509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.554509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:24.554509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:24.554509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:24.554509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.554509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:24.554509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:24.554509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:52:24.554509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:24.554509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:24.554509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.554509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:24.554834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:24.555121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:24.555465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:24.555930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:24.555930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:24.555930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:24.555930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:24.555930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:24.555930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.555930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:24.555930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:24.555930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.555930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:24.555930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.555930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:24.561675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:24.561813 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:24.561897 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:24.562005 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:24.562968 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:24.562968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.562968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:24.562968 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.562968 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:24.562968 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:24.562968 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.562968 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:24.562968 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.562968 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:24.563309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:24.563542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:24.563542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:24.563542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.563542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:24.563542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:24.563542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.563542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:24.563542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.563542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:24.563840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.563840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:24.563840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:24.563840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:24.563840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.563840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:24.563840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:24.563840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:24.563840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:24.563840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.563840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:24.564235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:24.564687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:24.565153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:24.565941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:24.565941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:24.565941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:24.565941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:24.565941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.565941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:24.565941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:24.565941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.565941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:24.565941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.565941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:24.569894 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:24.570048 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:24.570152 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:24.570324 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:24.570607 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:24.570793 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:24.570885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.571210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:24.571210 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.571210 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:24.571210 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:24.571210 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.571210 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:24.571210 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.571210 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:24.571895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.571895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:24.571895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:24.571895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:24.571895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.571895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:24.571895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:24.571895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.571895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:24.571895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:24.571895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:24.571895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.572097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:24.572310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.572310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:24.572632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:24.573061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:24.573363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:24.573363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:24.573363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:29.045430 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:29.046425 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:29.046571 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:29.047204 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:29.080231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:29.080464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.081062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:29.081062 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.081062 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:29.081062 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:29.081062 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.081062 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:29.081062 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.081062 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:29.082138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.082138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:29.082138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:29.082138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:29.082138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.082138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:29.082138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:29.082138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.082138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:29.082138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:29.082138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:29.082138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.082659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:29.082932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.082932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:29.083296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:29.083714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.084030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:29.084030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.084030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:29.538472 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:29.538860 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:29.538966 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:29.539590 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:29.539786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.540226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:29.540226 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.540226 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:29.540226 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:29.540226 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.540226 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:29.540226 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.540226 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:29.540942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.540942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:29.540942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:29.540942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:29.540942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.540942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:29.540942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:29.540942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.540942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:29.540942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:29.540942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:29.540942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.541596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:29.541833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:29.542115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.542500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:29.542500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:29.542500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.542500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:29.542500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.542500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:29.551942 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:29.552196 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:29.552302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:29.552410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:52:29.553230 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:29.553230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.553370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:29.553370 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.553370 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:29.553370 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:29.553370 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.553370 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:29.553370 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.553370 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:29.553946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:29.554291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:29.554291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:29.554291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.554291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:29.554291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:29.554291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.554291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:29.554291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.554291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:29.554542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.554542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:29.554542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:29.554542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:29.554542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.554542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:29.554542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:29.554542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:52:29.554542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:29.554542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:29.554542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.554542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:29.554805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:29.555034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.555034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:29.555034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.555034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:29.555202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:29.555487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:29.555823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.555823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:29.555823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:29.555823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.555823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:29.555823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.555823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:29.561948 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:29.562451 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:29.562550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:29.562673 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:29.563556 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:29.563556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.563719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:29.563719 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.563719 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:29.563719 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:29.563719 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.563719 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:29.563719 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.563719 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:29.564384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:29.564804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:29.564804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:29.564804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.564804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:29.564804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:29.564804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.564804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:29.564804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.564804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:29.565038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.565038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:29.565038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:29.565038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:29.565038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.565038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:29.565038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:29.565038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:29.565038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:29.565038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.565038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:29.565379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:29.565663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.565663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:29.565663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.565663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.565831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:29.566200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:29.566610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.566610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:29.566610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.566610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:29.570209 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:29.570456 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:29.570571 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:29.570721 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:29.571618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:29.571618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:29.571618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.571729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:29.571729 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.571729 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:29.571729 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:29.571729 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.571729 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:29.571729 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.571729 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:29.572446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.572446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:29.572446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:29.572446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:29.572446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.572446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:29.572446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:29.572446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.572446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:29.572446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:29.572446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:29.572446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.572826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:29.573160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.573160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:29.573250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:29.573688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:29.574174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:29.574174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:29.574174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:34.046490 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:34.047032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:34.047032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:34.047559 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:34.073381 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:34.073381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.073570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:34.073570 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.073570 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:34.073570 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:34.073570 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.073570 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:34.073570 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.073570 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:34.074325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.074325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:34.074325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:34.074325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:34.074325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.074325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:34.074325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:34.074325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.074325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:34.074325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:34.074325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:34.074325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.074582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:34.074809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.074809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:34.075134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:34.075717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.076029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:34.076029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.076029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:34.538932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:34.539321 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:34.539432 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:34.540491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:34.540491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.540718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:34.540718 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.540718 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:34.540718 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:34.540718 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.540718 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:34.540718 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.540718 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:34.541694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.541694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:34.541694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:34.541694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:34.541694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.541694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:34.541694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:34.541694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.541694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:34.541694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:34.541694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:34.541694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.541906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:34.542109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.542109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:34.542872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:34.543144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.543464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:34.543464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.543464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:34.552429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:34.552729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:34.552729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:34.554062 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:52:34.554062 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:34.554062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.554394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:34.554394 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.554394 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:34.554394 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:34.554394 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.554394 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:34.554394 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.554394 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:34.554819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:34.555332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:34.555332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:34.555332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.555332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:34.555332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:34.555332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.555332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:34.555332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.555332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:34.555616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.555616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:34.555616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:34.555616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:34.555616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.555616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:34.555616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:34.555616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:52:34.555616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:34.555616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:34.555616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.555616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:34.555899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:34.556281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:34.556642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:34.557094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:34.557094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:34.557094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:34.557094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:34.557094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:34.557094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.557094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:34.557094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:34.557094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.557094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:34.557094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.557094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:34.562580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:34.562954 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:34.563071 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:34.563161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:34.564025 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:34.564025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.564202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:34.564202 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.564202 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:34.564202 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:34.564202 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.564202 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:34.564202 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.564202 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:34.564817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:34.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:34.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:34.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:34.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:34.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:34.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.565160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:34.565667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.565667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:34.565667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:34.565667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:34.565667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.565667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:34.565667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:34.565667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:34.565667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:34.565667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.565667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:34.566245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:34.566616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:34.566865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:34.567151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:34.567151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:34.567151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:34.567151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:34.567151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.567151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:34.567151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:34.567151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.567151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:34.567151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.567151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:34.570661 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:34.570932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:34.571037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:34.571179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:34.572066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:34.572066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:34.572066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.572276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:34.572276 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.572276 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:34.572276 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:34.572276 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.572276 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:34.572276 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.572276 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:34.572921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.572921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:34.572921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:34.572921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:34.572921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.572921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:34.572921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:34.572921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.572921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:34.572921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:34.572921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:34.572921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.573273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:34.573638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.573638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:34.574075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:34.574858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:34.575608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:34.575608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:34.575608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:39.046895 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:39.047396 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:39.047396 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:39.047989 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:39.072201 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:39.072201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.072447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:39.072447 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.072447 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:39.072447 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:39.072447 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.072447 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:39.072447 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.072447 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:39.073020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.073020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:39.073020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:39.073020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:39.073020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.073020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:39.073020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:39.073020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.073020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:39.073020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:39.073020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:39.073020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.073273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:39.073712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:39.073967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.074287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:39.074287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:39.074287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.074287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:39.074287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.074287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:39.539367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:39.539810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:39.539944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:39.540879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:39.540879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.541039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:39.541039 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.541039 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:39.541039 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:39.541039 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.541039 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:39.541039 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.541039 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:39.541970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.541970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:39.541970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:39.541970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:39.541970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.541970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:39.541970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:39.541970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.541970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:39.541970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:39.541970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:39.541970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.542252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:39.542500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.542500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:39.543074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:39.543428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.543759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:39.543759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.543759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:39.552835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:39.553199 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:39.553365 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:39.553508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:52:39.554273 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:39.554273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.554484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:39.554484 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.554484 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:39.554484 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:39.554484 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.554484 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:39.554484 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.554484 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:39.555203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:39.555603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:39.555603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:39.555603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.555603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:39.555603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:39.555603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.555603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:39.555603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.555603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:39.556111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.556111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:39.556111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:39.556111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:39.556111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.556111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:39.556111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:39.556111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:52:39.556111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:39.556111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:39.556111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.556111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:39.556373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:39.556609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.556609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:39.556609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.556609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:39.557289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:39.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:39.557876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.557876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:39.557876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:39.557876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.557876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:39.557876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.557876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:39.562983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:39.563300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:39.563433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:39.563599 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:39.563914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:39.564108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.564557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:39.564557 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.564557 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:39.564557 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:39.564557 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.564557 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:39.564557 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.564557 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:39.565409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:39.565684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:39.565684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:39.565684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.565684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:39.565684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:39.565684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.565684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:39.565684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.565684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:39.566402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.566402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:39.566402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:39.566402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:39.566402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.566402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:39.566402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:39.566402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:39.566402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:39.566402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.566402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:39.566640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:39.566844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.566844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:39.566844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.566844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.567613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:39.567965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:39.568417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.568417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:39.568417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.568417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:39.571021 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:39.571328 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:39.571473 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:39.571685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:39.572068 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:39.572283 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:39.572367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.572879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:39.572879 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.572879 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:39.572879 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:39.572879 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.572879 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:39.572879 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.572879 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:39.573888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.573888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:39.573888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:39.573888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:39.573888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.573888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:39.573888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:39.573888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.573888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:39.573888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:39.573888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:39.573888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:39.574346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.574346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:39.574983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:39.575447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:39.575792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:39.575792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:39.575792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:44.047424 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:44.048056 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:44.048056 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:44.048336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:44.078034 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:44.078443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.079150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:44.079150 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.079150 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:44.079150 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:44.079150 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.079150 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:44.079150 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.079150 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:44.079934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.079934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:44.079934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:44.079934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:44.079934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.079934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:44.079934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:44.079934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.079934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:44.079934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:44.079934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:44.079934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.080173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:44.080397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.080397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:44.080695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:44.081465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.081851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:44.081851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.081851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:44.539803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:44.540047 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:44.540142 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:44.541143 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:44.541143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.541283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:44.541283 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.541283 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:44.541283 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:44.541283 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.541283 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:44.541283 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.541283 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:44.542060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.542060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:44.542060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:44.542060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:44.542060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.542060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:44.542060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:44.542060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.542060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:44.542060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:44.542060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:44.542060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.542494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:44.542719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.542719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:44.542881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:44.543363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.543989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:44.543989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.543989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:44.553150 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:44.553335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:44.553417 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:44.553498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:52:44.554249 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:44.554249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.554661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:44.554661 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.554661 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:44.554661 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:44.554661 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.554661 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:44.554661 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.554661 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:44.555228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:44.555596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:44.555596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:44.555596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.555596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:44.555596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:44.555596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.555596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:44.555596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.555596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:44.555806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.555806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:44.555806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:44.555806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:44.555806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.555806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:44.555806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:44.555806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:52:44.555806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:44.555806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:44.555806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.555806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:44.556108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:44.556442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.556442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:44.556442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.556442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:44.556701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:44.558367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:44.558970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.558970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:44.558970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:44.558970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.558970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:44.558970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.558970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:44.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:44.563428 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:44.563523 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:44.563623 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:44.563982 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:44.564236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.564683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:44.564683 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.564683 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:44.564683 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:44.564683 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.564683 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:44.564683 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.564683 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:44.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:44.565647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:44.565647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:44.565647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.565647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:44.565647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:44.565647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.565647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:44.565647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.565647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:44.565867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.565867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:44.565867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:44.565867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:44.565867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.565867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:44.565867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:44.565867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:44.565867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:44.565867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.565867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:44.566134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:44.566485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.566485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:44.566485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.566485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.566956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:44.567838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:44.568822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.568822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:44.568822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.568822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:44.571313 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:44.571471 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:44.571557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:44.571680 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:44.572019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:44.572240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:44.572356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.572843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:44.572843 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.572843 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:44.572843 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:44.572843 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.572843 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:44.572843 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.572843 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:44.573468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.573468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:44.573468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:44.573468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:44.573468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.573468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:44.573468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:44.573468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.573468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:44.573468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:44.573468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:44.573468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.573787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:44.574094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.574094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:44.574149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:44.574543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:44.574979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:44.574979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:44.574979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:49.047849 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:49.048608 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:49.048608 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:49.048798 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:49.074920 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:49.075126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.075708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:49.075708 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.075708 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:49.075708 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:49.075708 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.075708 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:49.075708 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.075708 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:49.076429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.076429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:49.076429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:49.076429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:49.076429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.076429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:49.076429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:49.076429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.076429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:49.076429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:49.076429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:49.076429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.077291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:49.077917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:49.078647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.079127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:49.079127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:49.079127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.079127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:49.079127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.079127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:49.540273 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:49.540705 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:49.540837 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:49.541506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:49.541704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.542204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:49.542204 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.542204 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:49.542204 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:49.542204 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.542204 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:49.542204 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.542204 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:49.543316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.543316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:49.543316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:49.543316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:49.543316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.543316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:49.543316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:49.543316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.543316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:49.543316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:49.543316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:49.543316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.543557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:49.543759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.543759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:49.544498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:49.544800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.545137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:49.545137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.545137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:49.553647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:49.553882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:49.554071 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:49.554071 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:52:49.554555 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:49.554746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.555178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:49.555178 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.555178 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:49.555178 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:49.555178 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.555178 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:49.555178 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.555178 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:49.555815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:49.556112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:49.556112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:49.556112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.556112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:49.556112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:49.556112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.556112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:49.556112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.556112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:49.556571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.556571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:49.556571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:49.556571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:49.556571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.556571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:49.556571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:49.556571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:52:49.556571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:49.556571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:49.556571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.556571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:49.556902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:49.557133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.557133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:49.557133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.557133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:49.557556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:49.558657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:49.559330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.559330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:49.559330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:49.559330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.559330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:49.559330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.559330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:49.563716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:49.563968 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:49.564040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:49.564113 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:49.564615 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:49.564860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.565394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:49.565394 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.565394 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:49.565394 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:49.565394 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.565394 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:49.565394 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.565394 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:49.566001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:49.566355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:49.566355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:49.566355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.566355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:49.566355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:49.566355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.566355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:49.566355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.566355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:49.566764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.566764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:49.566764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:49.566764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:49.566764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.566764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:49.566764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:49.566764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:49.566764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:49.566764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.566764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:49.567407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:49.567766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:49.568005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:49.568327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:49.568327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:49.568327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:49.568327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:49.568327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.568327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:49.568327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:49.568327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.568327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:49.568327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.568327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:49.571566 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:49.571740 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:49.571824 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:49.571947 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:49.572360 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:49.572558 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:49.572649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.573119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:49.573119 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.573119 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:49.573119 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:49.573119 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.573119 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:49.573119 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.573119 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:49.573744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.573744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:49.573744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:49.573744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:49.573744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.573744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:49.573744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:49.573744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.573744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:49.573744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:49.573744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:49.573744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.573974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:49.574178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.574178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:49.574470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:49.575296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:49.575679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:49.575679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:49.575679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:54.048285 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:54.048761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:54.048761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:54.049312 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:54.077622 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:54.077975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.078708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:54.078708 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.078708 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:54.078708 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:54.078708 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.078708 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:54.078708 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.078708 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:54.080139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.080139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:54.080139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:54.080139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:54.080139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.080139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:54.080139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:54.080139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.080139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:54.080139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:54.080139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:54.080139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.080580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:54.080972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.080972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:54.082354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:54.082785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.083241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:54.083241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.083241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:54.540723 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:54.541426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:54.541426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:54.541765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:54.541944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:54.543881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:54.544227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.544478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:54.544759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:54.545077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:54.545077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:54.545077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:54.545077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.545077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:54.545077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:54.545077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.545077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:54.545077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.545077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:54.554043 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:54.554802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:54.554802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:54.554802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:52:54.555176 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:54.555510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.557918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:54.558492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:54.558738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.558967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:54.559170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:54.559442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:54.559722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:54.559722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:54.559722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.559722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:54.559722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:54.559722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.559722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:54.559722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.559722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:54.564026 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:54.564599 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:54.564599 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:54.564599 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:54.565011 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:54.565319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.565902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:54.565902 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.565902 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:54.565902 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:54.565902 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.565902 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:54.565902 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.565902 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:54.568125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:54.568706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:54.569973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:54.569973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:54.569973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:54.569973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:54.569973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.569973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:54.569973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:54.569973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.569973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:54.569973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.569973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:54.569973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:54.570451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.571379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:54.572662 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:54.572662 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:54.572662 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:54.572662 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:54.572811 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:54.573082 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:54.573199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:54.575247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:54.575540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.575764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:54.576020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:54.576350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:54.576350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:54.576350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:54.576350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:54.576350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:54.576350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:54.576350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:54.576350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:54.576350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:54.576350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:59.048685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:59.049250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:59.049250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:59.049811 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:59.081149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:59.081459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.082176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:59.082176 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.082176 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:59.082176 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:59.082176 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.082176 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:59.082176 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.082176 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:59.083232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.083232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:59.083232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:59.083232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:59.083232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.083232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:59.083232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:59.083232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.083232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:59.083232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:59.083232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:59.083232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.083492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:59.083733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.083733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:59.084086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:59.084401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.084804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:59.084804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.084804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:59.541265 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:59.541735 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:59.541917 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:59.542553 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:59.542854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.543382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:59.543382 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.543382 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:59.543382 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:59.543382 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.543382 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:59.543382 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.543382 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:59.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:59.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:59.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:59.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:59.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:59.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:59.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:59.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:59.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.544740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:59.544945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.544945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:59.545548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:59.545879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.546181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:59.546181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.546181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:59.554636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:59.554946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:59.555073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:59.555177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:52:59.555638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:59.555820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.556310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:59.556310 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.556310 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:59.556310 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:59.556310 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.556310 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:59.556310 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.556310 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:59.556917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:59.557218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:59.557218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:59.557218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.557218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:59.557218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:59.557218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.557218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:59.557218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.557218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:59.557641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.557641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:59.557641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:59.557641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:59.557641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.557641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:59.557641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:59.557641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:52:59.557641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:59.557641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:59.557641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.557641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:59.558317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:59.558610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:59.558884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:59.559184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:59.559184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:59.559184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:59.559184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:59.559184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:59.559184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.559184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:59.559184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:59.559184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.559184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:59.559184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.559184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:59.564502 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:59.564866 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:59.564994 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:59.565139 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:59.565603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:59.565807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.566249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:59.566249 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.566249 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:59.566249 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:59.566249 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.566249 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:59.566249 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.566249 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:52:59.567014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:52:59.567365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:59.567365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:52:59.567365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.567365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:52:59.567365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:52:59.567365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.567365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:52:59.567365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.567365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:52:59.567774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.567774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:59.567774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:59.567774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:59.567774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.567774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:59.567774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:59.567774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:59.567774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:59.567774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.567774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:59.568413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:59.569032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:59.569289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:59.569655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:59.569655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:59.569655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:59.569655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:59.569655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.569655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:59.569655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:59.569655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.569655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:59.569655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.569655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:52:59.572270 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:52:59.572571 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:52:59.572673 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:52:59.572751 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:52:59.573163 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:59.573404 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:52:59.573502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.574005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:52:59.574005 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.574005 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:52:59.574005 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:52:59.574005 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.574005 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:52:59.574005 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.574005 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:52:59.574688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.574688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:52:59.574688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:59.574688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:52:59.574688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.574688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:59.574688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:59.574688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.574688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:59.574688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:59.574688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:52:59.574688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.575008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:52:59.575249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.575249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:52:59.575547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:52:59.576418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:52:59.576756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:52:59.576756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:52:59.576756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:04.049078 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:04.049754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:04.049754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:04.050022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:04.073891 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:04.074092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.074594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:04.074594 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.074594 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:04.074594 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:04.074594 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.074594 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:04.074594 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.074594 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:04.075309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.075309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:04.075309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:04.075309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:04.075309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.075309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:04.075309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:04.075309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.075309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:04.075309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:04.075309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:04.075309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.075578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:04.075780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.075780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:04.075980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:04.076285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.076654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:04.076654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.076654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:04.541728 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:04.542133 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:04.542276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:04.542740 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:04.542930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.543391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:04.543391 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.543391 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:04.543391 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:04.543391 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.543391 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:04.543391 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.543391 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:04.544164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.544164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:04.544164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:04.544164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:04.544164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.544164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:04.544164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:04.544164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.544164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:04.544164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:04.544164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:04.544164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.544618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:04.544968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:04.545409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.546030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:04.546030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:04.546030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.546030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:04.546030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.546030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:04.555059 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:04.555309 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:04.555384 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:04.555481 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:53:04.555866 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:04.556109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.556647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:04.556647 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.556647 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:04.556647 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:04.556647 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.556647 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:04.556647 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.556647 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:04.557205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:04.557630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:04.557630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:04.557630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.557630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:04.557630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:04.557630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.557630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:04.557630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.557630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:04.557896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.557896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:04.557896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:04.557896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:04.557896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.557896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:04.557896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:04.557896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:53:04.557896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:04.557896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:04.557896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.557896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:04.558266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:04.558662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:04.559063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:04.559607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:04.559607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:04.559607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:04.559607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:04.559607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:04.559607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.559607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:04.559607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:04.559607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.559607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:04.559607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.559607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:04.564866 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:04.565155 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:04.565268 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:04.565363 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:04.565706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:04.565897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.566318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:04.566318 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.566318 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:04.566318 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:04.566318 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.566318 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:04.566318 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.566318 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:04.566856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:04.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:04.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:04.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:04.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:04.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:04.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:04.567562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.567562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:04.567562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:04.567562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:04.567562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.567562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:04.567562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:04.567562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:04.567562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:04.567562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.567562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:04.567870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:04.568272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:04.568684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:04.569272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:04.569272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:04.569272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:04.569272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:04.569272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.569272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:04.569272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:04.569272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.569272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:04.569272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.569272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:04.572623 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:04.572862 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:04.572967 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:04.573136 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:04.573443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:04.573657 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:04.573741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.574159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:04.574159 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.574159 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:04.574159 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:04.574159 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.574159 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:04.574159 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.574159 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:04.575074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.575074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:04.575074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:04.575074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:04.575074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.575074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:04.575074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:04.575074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.575074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:04.575074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:04.575074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:04.575074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.575434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:04.575797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.575797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:04.576140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:04.576599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:04.577126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:04.577126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:04.577126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:09.049668 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:09.050261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:09.050261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:09.051019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:09.080032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:09.080238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.080706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:09.080706 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.080706 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:09.080706 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:09.080706 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.080706 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:09.080706 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.080706 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:09.081418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.081418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:09.081418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:09.081418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:09.081418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.081418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:09.081418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:09.081418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.081418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:09.081418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:09.081418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:09.081418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.081746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:09.082069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.082069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:09.082324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:09.082619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.082913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:09.082913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.082913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:09.542307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:09.542715 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:09.542892 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:09.543500 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:09.543721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.544295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:09.544295 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.544295 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:09.544295 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:09.544295 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.544295 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:09.544295 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.544295 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:09.545290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.545290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:09.545290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:09.545290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:09.545290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.545290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:09.545290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:09.545290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.545290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:09.545290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:09.545290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:09.545290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.545640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:09.545990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.545990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:09.546454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:09.546890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.547413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:09.547413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.547413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:09.555470 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:09.555852 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:09.556041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:09.556186 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:53:09.556594 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:09.556801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.557346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:09.557346 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.557346 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:09.557346 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:09.557346 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.557346 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:09.557346 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.557346 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:09.558039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:09.558562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:09.558562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:09.558562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.558562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:09.558562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:09.558562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.558562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:09.558562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.558562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:09.558940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.558940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:09.558940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:09.558940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:09.558940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.558940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:09.558940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:09.558940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:53:09.558940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:09.558940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:09.558940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.558940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:09.559306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:09.559739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.559739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:09.559739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.559739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:09.560064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:09.560586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:09.561063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.561063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:09.561063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:09.561063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.561063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:09.561063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.561063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:09.565434 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:09.565620 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:09.565746 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:09.565879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:09.566438 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:09.566711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:09.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:09.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:09.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:09.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.567301 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:09.568198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:09.568694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:09.568694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:09.568694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.568694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:09.568694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:09.568694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.568694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:09.568694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.568694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:09.569233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.569233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:09.569233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:09.569233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:09.569233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.569233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:09.569233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:09.569233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:09.569233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:09.569233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.569233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:09.569645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:09.570067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.570067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:09.570067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.570067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:09.570989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:09.571518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.571518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:09.571518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.571518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:09.572985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:09.573171 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:09.573306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:09.573482 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:09.573877 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:09.574115 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:09.574229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.574778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:09.574778 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.574778 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:09.574778 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:09.574778 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.574778 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:09.574778 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.574778 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:09.575765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.575765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:09.575765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:09.575765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:09.575765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.575765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:09.575765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:09.575765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.575765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:09.575765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:09.575765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:09.575765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.576102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:09.576517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.576517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:09.576765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:09.577381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:09.577918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:09.577918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:09.577918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:14.050079 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:14.050620 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:14.050777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:14.051497 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:14.080286 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:14.080536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.080975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:14.080975 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.080975 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:14.080975 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:14.080975 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.080975 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:14.080975 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.080975 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:14.081784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.081784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:14.081784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:14.081784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:14.081784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.081784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:14.081784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:14.081784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.081784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:14.081784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:14.081784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:14.081784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.082141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:14.082524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.082524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:14.082993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:14.083840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.084642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:14.084642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.084642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:14.542731 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:14.542967 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:14.543126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:14.543675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:14.543857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.544321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:14.544321 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.544321 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:14.544321 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:14.544321 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.544321 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:14.544321 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.544321 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:14.545071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.545071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:14.545071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:14.545071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:14.545071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.545071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:14.545071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:14.545071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.545071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:14.545071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:14.545071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:14.545071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.545326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:14.545553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.545553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:14.546117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:14.546609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.546912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:14.546912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.546912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:14.555879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:14.556178 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:14.556178 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:14.556343 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:53:14.556998 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:14.557257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.557855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:14.557855 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.557855 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:14.557855 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:14.557855 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.557855 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:14.557855 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.557855 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:14.558541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:14.558986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:14.559389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:14.560206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:14.560206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:14.560206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:14.560206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:14.560206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:14.560206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.560206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:14.560206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:14.560206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.560206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:14.560206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.560206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:14.560994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:14.561432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:14.561918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.561918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:14.561918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:14.561918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.561918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:14.561918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.561918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:14.565872 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:14.566079 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:14.566175 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:14.566382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:14.566660 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:14.566923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.567456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:14.567456 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.567456 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:14.567456 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:14.567456 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.567456 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:14.567456 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.567456 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:14.568000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:14.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:14.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:14.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:14.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:14.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:14.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.568406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:14.569247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.569247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:14.569247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:14.569247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:14.569247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.569247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:14.569247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:14.569247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:14.569247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:14.569247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.569247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:14.570146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:14.570540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:14.570919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:14.571405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:14.571405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:14.571405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:14.571405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:14.571405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.571405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:14.571405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:14.571405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.571405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:14.571405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.571405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:14.573393 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:14.573608 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:14.573735 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:14.573932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:14.574267 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:14.574449 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:14.574520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.574952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:14.574952 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.574952 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:14.574952 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:14.574952 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.574952 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:14.574952 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.574952 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:14.575819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.575819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:14.575819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:14.575819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:14.575819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.575819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:14.575819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:14.575819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.575819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:14.575819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:14.575819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:14.575819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.576608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:14.577289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:14.577663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:14.578124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:14.578124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:14.578124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:14.578124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:14.578124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:14.578124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:19.050760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:19.051379 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:19.051421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:19.052289 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:19.053735 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=260 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 00:53:19.087923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:19.088103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.088572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:19.088572 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.088572 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:19.088572 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:19.088572 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.088572 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:19.088572 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.088572 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:19.089307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.089307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:19.089307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:19.089307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:19.089307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.089307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:19.089307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:19.089307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.089307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:19.089307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:19.089307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:19.089307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.089584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:19.089790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.089790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:19.090097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:19.090488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.090796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:19.090796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.090796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:19.543120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:19.543744 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:19.543891 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:19.544465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:19.544661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.545307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:19.545307 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.545307 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:19.545307 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:19.545307 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.545307 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:19.545307 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.545307 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:19.546131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.546131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:19.546131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:19.546131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:19.546131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.546131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:19.546131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:19.546131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.546131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:19.546131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:19.546131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:19.546131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.546551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:19.546906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.546906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:19.547247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:19.548127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.549228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:19.549228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.549228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:19.556221 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:19.556659 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:19.556772 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:19.556908 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:53:19.557358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:19.557578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.558128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:19.558128 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.558128 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:19.558128 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:19.558128 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.558128 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:19.558128 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.558128 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:19.558890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:19.559345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:19.559345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:19.559345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.559345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:19.559345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:19.559345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.559345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:19.559345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.559345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:19.559796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.559796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:19.559796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:19.559796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:19.559796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.559796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:19.559796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:19.559796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:53:19.559796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:19.559796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:19.559796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.559796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:19.560442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:19.561095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:19.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:19.561979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:19.561979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:19.561979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:19.561979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:19.561979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:19.561979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.561979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:19.561979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:19.561979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.561979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:19.561979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.561979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:19.566248 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:19.566674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:19.566781 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:19.566926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:19.567400 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:19.567636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.568170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:19.568170 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.568170 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:19.568170 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:19.568170 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.568170 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:19.568170 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.568170 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:19.568866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:19.569270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:19.569270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:19.569270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.569270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:19.569270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:19.569270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.569270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:19.569270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.569270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:19.569753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.569753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:19.569753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:19.569753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:19.569753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.569753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:19.569753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:19.569753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:19.569753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:19.569753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.569753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:19.570482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:19.570907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:19.571282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:19.571709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:19.571709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:19.571709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:19.571709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:19.571709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.571709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:19.571709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:19.571709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.571709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:19.571709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.571709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:19.573753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:19.574066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:19.574156 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:19.574316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:19.574665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:19.574906 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:19.575031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.575520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:19.575520 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.575520 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:19.575520 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:19.575520 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.575520 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:19.575520 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.575520 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:19.576097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.576097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:19.576097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:19.576097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:19.576097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.576097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:19.576097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:19.576097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.576097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:19.576097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:19.576097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:19.576097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.576341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:19.576542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.576542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:19.576805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:19.577436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:19.577982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:19.577982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:19.577982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:24.051168 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:24.051735 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:24.051735 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:24.052175 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:24.077807 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:24.078044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.078523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:24.078523 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.078523 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:24.078523 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:24.078523 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.078523 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:24.078523 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.078523 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:24.079256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.079256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:24.079256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:24.079256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:24.079256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.079256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:24.079256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:24.079256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.079256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:24.079256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:24.079256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:24.079256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.079513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:24.079738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.079738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:24.080016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:24.080917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.081360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:24.081360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.081360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:24.543555 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:24.543867 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:24.543958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:24.544606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:24.544810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.545266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:24.545266 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.545266 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:24.545266 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:24.545266 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.545266 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:24.545266 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.545266 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:24.546007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.546007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:24.546007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:24.546007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:24.546007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.546007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:24.546007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:24.546007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.546007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:24.546007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:24.546007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:24.546007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.546326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:24.546522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.546522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:24.546895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:24.547716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.548072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:24.548072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.548072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:24.556514 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:24.556703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:24.556785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:24.556861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:53:24.557335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:24.557504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.557960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:24.557960 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.557960 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:24.557960 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:24.557960 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.557960 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:24.557960 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.557960 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:24.558500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:24.558785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:24.558785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:24.558785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.558785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:24.558785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:24.558785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.558785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:24.558785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.558785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:24.559118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.559118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:24.559118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:24.559118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:24.559118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.559118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:24.559118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:24.559118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:53:24.559118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:24.559118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:24.559118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.559118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:24.559747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:24.560061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:24.560321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:24.560630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:24.560630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:24.560630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:24.560630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:24.560630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:24.560630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.560630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:24.560630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:24.560630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.560630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:24.560630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.560630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:24.566612 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:24.566852 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:24.566952 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:24.567090 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:24.567391 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:24.567644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:24.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:24.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:24.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:24.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.568067 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:24.568558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:24.568833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:24.568833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:24.568833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.568833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:24.568833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:24.568833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.568833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:24.568833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.568833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:24.569172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.569172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:24.569172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:24.569172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:24.569172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.569172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:24.569172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:24.569172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:24.569172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:24.569172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.569172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:24.569790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:24.570151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:24.570434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:24.570745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:24.570745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:24.570745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:24.570745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:24.570745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.570745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:24.570745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:24.570745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.570745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:24.570745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.570745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:24.574174 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:24.574346 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:24.574443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:24.574576 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:24.574887 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:24.575118 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:24.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.575707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:24.575707 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.575707 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:24.575707 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:24.575707 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.575707 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:24.575707 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.575707 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:24.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:24.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:24.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:24.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:24.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:24.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:24.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:24.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:24.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.576523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:24.576724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.576724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:24.576973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:24.577751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:24.578144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:24.578144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:24.578144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:29.051634 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:29.052108 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:29.052108 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:29.052682 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:29.077104 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:29.077259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.077697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:29.077697 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.077697 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:29.077697 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:29.077697 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.077697 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:29.077697 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.077697 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:29.078340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.078340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:29.078340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:29.078340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:29.078340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.078340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:29.078340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:29.078340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.078340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:29.078340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:29.078340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:29.078340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.078591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:29.078801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.078801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:29.079169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:29.079749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.080335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:29.080335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.080335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:29.544006 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:29.544647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:29.544880 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:29.545443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:29.545645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.546344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:29.546344 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.546344 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:29.546344 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:29.546344 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.546344 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:29.546344 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.546344 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:29.547005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.547005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:29.547005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:29.547005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:29.547005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.547005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:29.547005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:29.547005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.547005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:29.547005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:29.547005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:29.547005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.547334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:29.547663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.547663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:29.548053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:29.548761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.549515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:29.549515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.549515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:29.557037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:29.557281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:29.557411 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:29.557489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:53:29.557827 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:29.558027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.558457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:29.558457 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.558457 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:29.558457 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:29.558457 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.558457 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:29.558457 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.558457 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:29.558996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:29.559413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:29.559413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:29.559413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.559413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:29.559413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:29.559413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.559413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:29.559413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.559413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:29.559860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.559860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:29.559860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:29.559860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:29.559860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.559860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:29.559860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:29.559860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:53:29.559860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:29.559860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:29.559860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.559860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:29.560649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:29.561392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:29.561775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:29.562287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:29.562287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:29.562287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:29.562287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:29.562287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:29.562287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.562287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:29.562287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:29.562287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.562287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:29.562287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.562287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:29.567102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:29.567334 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:29.567434 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:29.567544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:29.567858 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:29.568063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.568499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:29.568499 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.568499 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:29.568499 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:29.568499 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.568499 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:29.568499 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.568499 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:29.569047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:29.569456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:29.569456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:29.569456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.569456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:29.569456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:29.569456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.569456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:29.569456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.569456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:29.569888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.569888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:29.569888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:29.569888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:29.569888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.569888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:29.569888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:29.569888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:29.569888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:29.569888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.569888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:29.570443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:29.571248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:29.571579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:29.572064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:29.572064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:29.572064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:29.572064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:29.572064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.572064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:29.572064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:29.572064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.572064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:29.572064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.572064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:29.574431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:29.574589 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:29.574657 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:29.574823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:29.575088 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:29.575327 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:29.575421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.575848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:29.575848 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.575848 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:29.575848 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:29.575848 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.575848 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:29.575848 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.575848 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:29.576436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.576436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:29.576436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:29.576436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:29.576436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.576436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:29.576436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:29.576436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.576436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:29.576436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:29.576436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:29.576436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.576762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:29.577137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:29.577573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:29.578050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:29.578050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:29.578050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:29.578050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:29.578050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:29.578050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:34.052025 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:34.052685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:34.052685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:34.053214 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:34.083208 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:34.083407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.083883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:34.083883 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.083883 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:34.083883 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:34.083883 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.083883 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:34.083883 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.083883 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:34.084876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.084876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:34.084876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:34.084876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:34.084876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.084876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:34.084876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:34.084876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.084876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:34.084876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:34.084876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:34.084876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.085161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:34.085439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.085439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:34.085969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:34.086301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.086673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:34.086673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.086673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:34.544654 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:34.544869 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:34.544999 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:34.545427 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:34.545588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.546031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:34.546031 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.546031 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:34.546031 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:34.546031 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.546031 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:34.546031 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.546031 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:34.546702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.546702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:34.546702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:34.546702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:34.546702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.546702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:34.546702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:34.546702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.546702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:34.546702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:34.546702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:34.546702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.546997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:34.547294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.547294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:34.547678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:34.548030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.548491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:34.548491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.548491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:34.557429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:34.557618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:34.557733 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:34.557924 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:53:34.558245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:34.558397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.558774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:34.558774 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.558774 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:34.558774 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:34.558774 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.558774 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:34.558774 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.558774 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:34.559318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:34.559746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:34.559746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:34.559746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.559746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:34.559746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:34.559746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.559746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:34.559746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.559746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:34.559954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.559954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:34.559954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:34.559954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:34.559954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.559954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:34.559954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:34.559954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:53:34.559954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:34.559954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:34.559954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.559954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:34.560179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:34.560528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:34.560812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:34.561177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:34.561177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:34.561177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:34.561177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:34.561177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:34.561177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.561177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:34.561177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:34.561177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.561177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:34.561177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.561177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:34.567638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:34.567930 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:34.568028 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:34.568132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:34.568416 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:34.568618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.569009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:34.569009 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.569009 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:34.569009 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:34.569009 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.569009 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:34.569009 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.569009 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:34.569499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:34.569949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:34.569949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:34.569949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.569949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:34.569949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:34.569949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.569949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:34.569949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.569949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:34.570236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.570236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:34.570236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:34.570236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:34.570236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.570236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:34.570236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:34.570236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:34.570236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:34.570236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.570236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:34.570578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:34.570966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:34.571378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:34.571930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:34.571930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:34.571930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:34.571930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:34.571930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.571930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:34.571930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:34.571930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.571930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:34.571930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.571930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:34.574813 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:34.575122 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:34.575228 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:34.575344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:34.575637 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:34.575789 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:34.575906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.576316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:34.576316 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.576316 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:34.576316 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:34.576316 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.576316 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:34.576316 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.576316 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:34.577009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.577009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:34.577009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:34.577009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:34.577009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.577009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:34.577009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:34.577009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.577009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:34.577009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:34.577009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:34.577009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.577426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:34.577839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.577839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:34.578046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:34.578390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:34.578701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:34.578701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:34.578701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:39.052451 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:39.052920 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:39.052920 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:39.053445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:39.078304 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:39.078595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.079242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:39.079242 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.079242 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:39.079242 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:39.079242 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.079242 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:39.079242 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.079242 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:39.079978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.079978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:39.079978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:39.079978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:39.079978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.079978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:39.079978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:39.079978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.079978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:39.079978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:39.079978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:39.079978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.080256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:39.080489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.080489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:39.080720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:39.081472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.082133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:39.082133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.082133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:39.545042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:39.545281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:39.545383 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:39.545817 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:39.545996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.546417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:39.546417 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.546417 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:39.546417 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:39.546417 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.546417 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:39.546417 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.546417 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:39.547058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.547058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:39.547058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:39.547058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:39.547058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.547058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:39.547058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:39.547058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.547058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:39.547058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:39.547058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:39.547058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.547312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:39.547511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.547511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:39.547745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:39.548025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.548333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:39.548333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.548333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:39.557784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:39.558022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:39.558097 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:39.558166 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:53:39.558449 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:39.558616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.558998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:39.558998 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.558998 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:39.558998 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:39.558998 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.558998 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:39.558998 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.558998 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:39.559581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:39.560051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:39.560051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:39.560051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.560051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:39.560051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:39.560051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.560051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:39.560051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.560051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:39.560315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.560315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:39.560315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:39.560315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:39.560315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.560315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:39.560315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:39.560315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:53:39.560315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:39.560315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:39.560315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.560315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:39.560662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:39.561017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:39.561437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:39.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:39.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:39.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:39.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:39.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:39.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:39.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:39.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:39.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:39.568080 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:39.568277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:39.568382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:39.568534 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:39.568759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:39.568955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.569384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:39.569384 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.569384 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:39.569384 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:39.569384 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.569384 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:39.569384 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.569384 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:39.569882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:39.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:39.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:39.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:39.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:39.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:39.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:39.570545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.570545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:39.570545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:39.570545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:39.570545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.570545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:39.570545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:39.570545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:39.570545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:39.570545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.570545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:39.570845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:39.571285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:39.571689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:39.572202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:39.572202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:39.572202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:39.572202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:39.572202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.572202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:39.572202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:39.572202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.572202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:39.572202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.572202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:39.575242 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:39.575418 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:39.575516 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:39.575622 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:39.575891 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:39.576095 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:39.576203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.576552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:39.576552 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.576552 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:39.576552 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:39.576552 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.576552 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:39.576552 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.576552 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:39.577157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.577157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:39.577157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:39.577157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:39.577157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.577157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:39.577157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:39.577157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.577157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:39.577157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:39.577157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:39.577157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.577676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:39.578111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:39.578479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:39.578935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:39.578935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:39.578935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:39.578935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:39.578935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:39.578935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:44.052797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:44.053458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:44.053458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:44.053845 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:44.078302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:44.078426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.079016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:44.079016 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.079016 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:44.079016 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:44.079016 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.079016 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:44.079016 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.079016 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:44.080114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.080114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:44.080114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:44.080114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:44.080114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.080114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:44.080114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:44.080114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.080114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:44.080114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:44.080114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:44.080114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.080553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:44.080988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.080988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:44.081209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:44.081687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.082297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:44.082297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.082297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:44.545520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:44.545880 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:44.546045 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:44.546556 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:44.546726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.547403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:44.547403 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.547403 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:44.547403 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:44.547403 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.547403 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:44.547403 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.547403 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:44.548756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.548756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:44.548756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:44.548756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:44.548756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.548756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:44.548756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:44.548756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.548756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:44.548756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:44.548756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:44.548756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.549083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:44.549451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.549451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:44.550308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:44.550740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.551247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:44.551247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.551247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:44.558358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:44.558585 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:44.558742 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:44.558861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:53:44.559337 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:44.561070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.561810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:44.562587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.562587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:44.562587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:44.562587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:44.562587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.562587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:44.562587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:44.562587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:53:44.562587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:44.562587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:44.562587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.562587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:44.562995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:44.563929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:44.564492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:44.565071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:44.565071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:44.565071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:44.565071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:44.565071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:44.565071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.565071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:44.565071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:44.565071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.565071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:44.565071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.565071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:44.568398 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:44.568722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:44.568806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:44.569001 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:44.569494 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:44.569763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:44.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:44.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:44.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:44.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.570329 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:44.571742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:44.572602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:44.572602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:44.572602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.572602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:44.572602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:44.572602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.572602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:44.572602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.572602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:44.573035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.573035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:44.573035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:44.573035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:44.573035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.573035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:44.573035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:44.573035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:44.573035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:44.573035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.573035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:44.573386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:44.573798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.573798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:44.573798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.573798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.574699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:44.575293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:44.575810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.575810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:44.575810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.575810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:44.575810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:44.575978 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:44.576227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:44.576297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:44.576633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:44.576903 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:44.577019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.577660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:44.577660 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.577660 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:44.577660 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:44.577660 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.577660 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:44.577660 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.577660 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:44.578615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.578615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:44.578615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:44.578615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:44.578615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.578615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:44.578615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:44.578615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.578615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:44.578615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:44.578615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:44.578615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.579025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:44.579451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.579451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:44.579692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:44.580217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:44.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:44.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:44.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:49.053301 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:49.053732 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:49.053732 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:49.054216 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:49.085119 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:49.085478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.085864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:49.085864 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.085864 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:49.085864 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:49.085864 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.085864 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:49.085864 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.085864 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:49.086836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.086836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:49.086836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:49.086836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:49.086836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.086836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:49.086836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:49.086836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.086836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:49.086836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:49.086836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:49.086836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.087160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:49.087449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.087449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:49.087831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:49.088153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.088556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:49.088556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.088556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:49.545934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:49.546166 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:49.546312 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:49.546839 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:49.547033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.547507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:49.547507 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.547507 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:49.547507 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:49.547507 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.547507 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:49.547507 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.547507 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:49.548167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.548167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:49.548167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:49.548167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:49.548167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.548167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:49.548167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:49.548167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.548167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:49.548167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:49.548167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:49.548167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.548496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:49.548807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.548807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:49.548992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:49.549393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.549722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:49.549722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.549722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:49.558732 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:49.559005 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:49.559113 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:49.559241 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:53:49.559618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:49.559833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.560396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:49.560396 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.560396 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:49.560396 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:49.560396 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.560396 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:49.560396 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.560396 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:49.561485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:49.561980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:49.561980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:49.561980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.561980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:49.561980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:49.561980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.561980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:49.561980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.561980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:49.562405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.562405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:49.562405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:49.562405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:49.562405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.562405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:49.562405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:49.562405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:53:49.562405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:49.562405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:49.562405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.562405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:49.562637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:49.562890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.562890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:49.562890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.562890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:49.563286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:49.563572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:49.563842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.563842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:49.563842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:49.563842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.563842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:49.563842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.563842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:49.568771 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:49.569325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:49.569397 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:49.569650 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:49.570032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:49.570504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.570590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:49.570590 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.570590 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:49.570590 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:49.570590 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.570590 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:49.570590 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.570590 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:49.571271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:49.571781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:49.571781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:49.571781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.571781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:49.571781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:49.571781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.571781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:49.571781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.571781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:49.572099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.572099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:49.572099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:49.572099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:49.572099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.572099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:49.572099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:49.572099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:49.572099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:49.572099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.572099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:49.572522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:49.572806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:49.573067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:49.573407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:49.573407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:49.573407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:49.573407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:49.573407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.573407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:49.573407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:49.573407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.573407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:49.573407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.573407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:49.576093 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:49.576226 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:49.576278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:49.576420 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:49.576672 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:49.576820 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:49.576915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.577298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:49.577298 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.577298 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:49.577298 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:49.577298 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.577298 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:49.577298 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.577298 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:49.578159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.578159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:49.578159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:49.578159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:49.578159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.578159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:49.578159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:49.578159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.578159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:49.578159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:49.578159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:49.578159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.578462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:49.579220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:49.579504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:49.579941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:49.579941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:49.579941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:54.053681 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:54.054357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:54.054357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:54.054555 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:54.084129 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:54.084270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.084670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:54.084670 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.084670 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:54.084670 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:54.084670 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.084670 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:54.084670 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.084670 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:54.085276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.085276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:54.085276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:54.085276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:54.085276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.085276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:54.085276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:54.085276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.085276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:54.085276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:54.085276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:54.085276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.085550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:54.085817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.085817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:54.086031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:54.086395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.086718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:54.086718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.086718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:54.546314 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:54.546637 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:54.546759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:54.548789 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:54.548789 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:54.548789 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:54.548789 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:54.548789 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce8f4a32-94cc-4682-94cb-97f17b86edff, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:54.548958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:54.548958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:54.548958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.548958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:54.548958 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.548958 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:54.548958 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:54.548958 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.548958 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:54.548958 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.548958 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:54.549631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.549631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:54.549631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:54.549631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:54.549631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.549631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:54.549631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:54.549631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.549631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:54.549631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:54.549631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:54.549631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.549945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:54.550175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.550175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:54.550427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:54.550741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.551254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:54.551254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.551254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:54.559064 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:54.559260 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:54.559334 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:54.559436 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:53:54.560143 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:54.560143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.560261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:54.560261 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.560261 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:54.560261 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:54.560261 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.560261 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:54.560261 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.560261 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:54.560793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:54.561108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:54.561108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:54.561108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.561108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:54.561108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:54.561108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.561108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:54.561108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.561108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:54.561352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.561352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:54.561352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:54.561352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:54.561352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.561352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:54.561352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:54.561352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:53:54.561352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:54.561352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:54.561352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.561352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:54.561596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:54.561880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.561880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:54.561880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.561880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:54.562058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:54.562381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:54.562671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.562671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:54.562671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:54.562671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.562671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:54.562671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.562671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:54.569445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:54.569719 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:54.569797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:54.569911 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:54.570676 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:54.570676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.570824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:54.570824 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.570824 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:54.570824 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:54.570824 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.570824 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:54.570824 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.570824 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:54.571290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:54.571568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:54.571568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:54.571568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.571568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:54.571568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:54.571568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.571568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:54.571568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.571568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:54.572266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.572266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:54.572266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:54.572266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:54.572266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.572266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:54.572266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:54.572266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:54.572266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:54.572266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.572266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:54.572560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:54.572804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:54.573091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:54.573421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:54.573421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:54.573421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:54.573421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:54.573421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.573421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:54.573421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:54.573421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.573421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:54.573421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.573421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:54.576263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:54.576441 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:54.576541 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:54.576698 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:54.577432 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:54.577432 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:54.577432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.577763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:54.577763 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.577763 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:54.577763 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:54.577763 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.577763 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:54.577763 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.577763 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:54.578544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.578544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:54.578544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:54.578544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:54.578544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.578544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:54.578544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:54.578544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.578544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:54.578544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:54.578544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:54.578544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:54.579294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.579294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:54.579689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:54.580175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:54.580813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:54.580813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:54.580813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:59.054066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:59.054596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:59.054596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:59.055172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:59.095837 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.095837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.095837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:59.095837 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.095837 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:59.095837 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:59.095837 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.095837 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:59.095837 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.095837 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:59.096809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.096809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:59.096809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:59.096809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:59.096809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.096809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:59.096809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:59.096809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.096809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:59.096809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:59.096809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:59.096809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:59.097738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.097738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:59.097941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:59.098433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.098985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:59.098985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.098985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:59.548060 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:59.549137 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:59.549416 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:59.550595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.550789 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce8f4a32-94cc-4682-94cb-97f17b86edff, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.550892 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.551206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.551206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.551206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.551332 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.551385 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.551518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.552208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:59.552208 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.552208 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:59.552208 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:59.552208 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.552208 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:59.552208 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.552208 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:59.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:59.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:59.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:59.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:59.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:59.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:59.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:59.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:59.553204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.553607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:59.554009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.554009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:59.556375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:59.557249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.558324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:59.558324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.558324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:59.559637 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:59.560234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:59.560354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:59.560479 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:53:59.560827 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.561041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.561692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:59.561692 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.561692 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:59.561692 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:59.561692 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.561692 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:59.561692 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.561692 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:59.563004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:59.563593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:59.563593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:59.563593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.563593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:59.563593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:59.563593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.563593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:59.563593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.563593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:59.564685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.564685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:59.564685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:59.564685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:59.564685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.564685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:59.564685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:59.564685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:53:59.564685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:59.564685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:59.564685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.564685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:59.565102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:59.565608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.565608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:59.565608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.565608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:59.566361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:59.566938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:59.567450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.567450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:59.567450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:59.567450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.567450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:59.567450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.567450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:59.569982 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:59.570044 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:59.570180 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:59.570313 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:59.570754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.571343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.571804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:59.571804 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.571804 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:59.571804 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:59.571804 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.571804 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:59.571804 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.571804 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:53:59.572622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:53:59.572916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:59.572916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:53:59.572916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.572916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:53:59.572916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:53:59.572916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.572916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:53:59.572916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.572916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:53:59.573588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.573588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:59.573588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:59.573588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:59.573588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.573588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:59.573588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:59.573588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:59.573588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:59.573588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.573588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:59.573806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:59.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:59.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.574433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:59.574707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:59.575049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.575049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:59.575049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.575049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:53:59.576525 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:53:59.576843 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:53:59.577026 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:53:59.577291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:53:59.578169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.578169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:53:59.578408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:53:59.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:53:59.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:53:59.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:53:59.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.578921 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:53:59.579716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.579716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:53:59.579716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:59.579716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:53:59.579716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.579716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:59.579716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:59.579716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.579716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:59.579716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:59.579716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:53:59.579716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.580095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:53:59.580486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:53:59.580919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:53:59.581483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:53:59.581483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:53:59.581483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:53:59.581483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:53:59.581483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:53:59.581483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:04.054475 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:04.054996 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:04.054996 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:04.055553 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:04.090458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.090732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.091344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:04.091344 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.091344 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:04.091344 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:04.091344 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.091344 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:04.091344 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.091344 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:04.092169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.092169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:04.092169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:04.092169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:04.092169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.092169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:04.092169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:04.092169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.092169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:04.092169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:04.092169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:04.092169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.092417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:04.092626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.092626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:04.093063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:04.093389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.093693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:04.093693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.093693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:04.549327 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:04.549633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:04.549782 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:04.550944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.551239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.551423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.551502 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.551592 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce8f4a32-94cc-4682-94cb-97f17b86edff, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.551744 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.552176 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.552460 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.552668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.553407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:04.553407 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.553407 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:04.553407 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:04.553407 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.553407 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:04.553407 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.553407 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:04.554468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.554468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:04.554468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:04.554468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:04.554468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.554468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:04.554468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:04.554468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.554468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:04.554468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:04.554468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:04.554468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.554842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:04.555057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.555057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:04.555242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:04.555566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.555976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:04.555976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.555976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:04.560137 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:04.560320 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:04.560413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:04.560503 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:54:04.560960 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.561167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.561743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:04.561743 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.561743 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:04.561743 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:04.561743 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.561743 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:04.561743 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.561743 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:04.562294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:04.562618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:04.562618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:04.562618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.562618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:04.562618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:04.562618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.562618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:04.562618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.562618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:04.562845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.562845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:04.562845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:04.562845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:04.562845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.562845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:04.562845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:04.562845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:54:04.562845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:04.562845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:04.562845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.562845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:04.563091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:04.563379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.563379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:04.563379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.563379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:04.563496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:04.563776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:04.564232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.564232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:04.564232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:04.564232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.564232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:04.564232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.564232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:04.570171 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:04.570453 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:04.570522 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:04.570633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:04.571054 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.571322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.571790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:04.571790 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.571790 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:04.571790 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:04.571790 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.571790 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:04.571790 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.571790 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:04.572319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:04.572634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:04.572634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:04.572634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.572634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:04.572634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:04.572634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.572634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:04.572634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.572634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:04.573163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.573163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:04.573163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:04.573163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:04.573163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.573163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:04.573163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:04.573163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:04.573163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:04.573163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.573163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:04.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:04.573651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.573651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:04.573651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.573651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.574315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:04.574632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:04.574999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.574999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:04.574999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.574999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:04.577231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:04.577430 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:04.577568 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:04.577758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:04.578528 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.578967 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:04.579199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.579738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:04.579738 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.579738 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:04.579738 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:04.579738 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.579738 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:04.579738 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.579738 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:04.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:04.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:04.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:04.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:04.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:04.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:04.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:04.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:04.580788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.581059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:04.581343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.581343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:04.581760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:04.582136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:04.582519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:04.582519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:04.582519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:09.055118 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:09.055498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:09.055498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:09.056201 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:09.093433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.093755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.094200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:09.094200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.094200 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:09.094200 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:09.094200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.094200 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:09.094200 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.094200 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:09.094961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.094961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:09.094961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:09.094961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:09.094961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.094961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:09.094961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:09.094961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.094961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:09.094961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:09.094961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:09.094961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.095340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:09.095677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.095677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:09.096131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:09.096793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.097296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:09.097296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.097296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:09.549778 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:09.550243 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:09.550341 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:09.551382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.551562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.551688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.551809 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.551993 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce8f4a32-94cc-4682-94cb-97f17b86edff, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.553234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.553234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.553234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.553234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.553234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.553474 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f090e551-026c-4a92-b933-a403a0e0e5aa, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.553474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.554502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:09.554502 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.554502 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:09.554502 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:09.554502 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.554502 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:09.554502 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.554502 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:09.554864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.554864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:09.554864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:09.554864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:09.554864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.554864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:09.554864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:09.554864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.554864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:09.554864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:09.554864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:09.554864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.555726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:09.556377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.556377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:09.557850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:09.559866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.560384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:09.560384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.560384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:09.560886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:09.561873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:09.561873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:09.561873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:54:09.562646 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.564762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.565482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:09.565482 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.565482 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:09.565482 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:09.565482 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.565482 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:09.565482 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.565482 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:09.566260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:09.566651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:09.566651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:09.566651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.566651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:09.566651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:09.566651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.566651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:09.566651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.566651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:09.566861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.566861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:09.566861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:09.566861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:09.566861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.566861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:09.566861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:09.566861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:54:09.566861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:09.566861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:09.566861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.566861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:09.567118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:09.567386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.567386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:09.567386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.567386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:09.567552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:09.567887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:09.568163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.568163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:09.568163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:09.568163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.568163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:09.568163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.568163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:09.570665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:09.570973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:09.571069 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:09.571198 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:09.571584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.572241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.572506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:09.572506 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.572506 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:09.572506 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:09.572506 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.572506 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:09.572506 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.572506 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:09.573182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:09.573531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:09.573531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:09.573531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.573531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:09.573531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:09.573531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.573531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:09.573531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.573531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:09.573761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.573761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:09.573761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:09.573761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:09.573761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.573761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:09.573761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:09.573761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:09.573761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:09.573761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.573761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:09.574031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:09.574288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.574288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:09.574288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.574288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.574563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:09.574963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:09.575426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.575426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:09.575426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.575426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:09.577645 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:09.578005 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:09.578122 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:09.578339 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:09.579530 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.579806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:09.580133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.580653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:09.580653 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.580653 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:09.580653 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:09.580653 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.580653 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:09.580653 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.580653 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:09.581611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.581611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:09.581611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:09.581611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:09.581611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.581611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:09.581611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:09.581611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.581611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:09.581611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:09.581611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:09.581611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:09.582163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.582163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:09.582566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:09.582914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:09.583244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:09.583244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:09.583244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:14.055486 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:14.056008 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:14.056008 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:14.056760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:14.099355 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.099843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.100145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:14.100145 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.100145 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:14.100145 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:14.100145 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.100145 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:14.100145 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.100145 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:14.100903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.100903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:14.100903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:14.100903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:14.100903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.100903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:14.100903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:14.100903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.100903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:14.100903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:14.100903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:14.100903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.101318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:14.101639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:14.101908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.102238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:14.102238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:14.102238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.102238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:14.102238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.102238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:14.550261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:14.552324 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:14.552324 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:14.552585 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.552754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.552878 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.552984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.553092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce8f4a32-94cc-4682-94cb-97f17b86edff, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.553169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.553546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.553698 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.553866 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-f090e551-026c-4a92-b933-a403a0e0e5aa, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.554001 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.554142 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.554298 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.554431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-8838ae75-8094-46bb-aad0-ea97037039d3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.554582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.555102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:14.555102 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.555102 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:14.555102 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:14.555102 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.555102 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:14.555102 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.555102 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:14.556000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.556000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:14.556000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:14.556000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:14.556000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.556000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:14.556000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:14.556000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.556000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:14.556000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:14.556000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:14.556000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.556432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:14.556843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.556843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:14.557139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:14.557729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.558253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:14.558253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.558253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:14.561420 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:14.561777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:14.561777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:14.561913 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:54:14.562264 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.562471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.562901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:14.562901 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.562901 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:14.562901 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:14.562901 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.562901 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:14.562901 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.562901 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:14.563397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:14.564813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:14.565286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:14.565531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:14.565531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:14.565531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:14.565531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:14.565531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:14.565531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.565531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:14.565531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:14.565531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.565531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:14.565531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.565531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:14.565740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:14.566112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:14.566545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.566545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:14.566545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:14.566545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.566545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:14.566545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.566545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:14.571127 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:14.571442 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:14.571497 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:14.571569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:14.571885 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:14.573714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.574057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:14.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:14.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:14.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:14.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:14.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:14.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:14.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:14.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:14.574583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:14.574871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:14.575271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:14.576266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:14.576266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:14.576266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:14.576266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:14.576266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.576266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:14.576266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:14.576266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.576266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:14.576266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.576266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:14.577983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:14.578179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:14.578275 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:14.578402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:14.578658 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.578839 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:14.579077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.579459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:14.579459 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.579459 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:14.579459 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:14.579459 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.579459 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:14.579459 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.579459 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:14.580057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.580057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:14.580057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:14.580057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:14.580057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.580057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:14.580057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:14.580057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.580057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:14.580057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:14.580057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:14.580057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.580443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:14.581119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:14.581863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:14.582315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:14.582315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:14.582315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:14.582315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:14.582315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:14.582315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:19.057710 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:19.058253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:19.058253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:19.058852 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:19.062041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=240 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 00:54:19.099515 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.099699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.100196 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:19.100196 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.100196 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:19.100196 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:19.100196 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.100196 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:19.100196 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.100196 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:19.100909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.100909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:19.100909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:19.100909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:19.100909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.100909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:19.100909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:19.100909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.100909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:19.100909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:19.100909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:19.100909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.101214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:19.101467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.101467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:19.101613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:19.101931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.102243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:19.102243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.102243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:19.552128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:19.553668 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:19.555091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:19.557696 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.557900 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.558071 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.558245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.558380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce8f4a32-94cc-4682-94cb-97f17b86edff, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.558484 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.558596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.558679 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.558794 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.558917 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.559019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f090e551-026c-4a92-b933-a403a0e0e5aa, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.559112 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.559252 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-8838ae75-8094-46bb-aad0-ea97037039d3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.559353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.559990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:19.559990 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.559990 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:19.559990 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:19.559990 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.559990 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:19.559990 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.559990 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:19.562208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.562208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:19.562208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:19.562208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:19.562208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.562208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:19.562208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:19.562208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.562208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:19.562208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:19.562208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:19.562208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.562578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:19.562914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.562914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:19.562914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:19.566015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:19.566561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.567072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:19.567072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.567072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:19.567072 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:19.567072 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:19.567072 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:54:19.567072 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.567072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.567990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:19.567990 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.567990 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:19.567990 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:19.567990 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.567990 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:19.567990 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.567990 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:19.568998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:19.569845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:19.569845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:19.569845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.569845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:19.569845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:19.569845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.569845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:19.569845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.569845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:19.570108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.570108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:19.570108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:19.570108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:19.570108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.570108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:19.570108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:19.570108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:54:19.570108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:19.570108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:19.570108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.570108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:19.570428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:19.570761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.570761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:19.570761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.570761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:19.571973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:19.573076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:19.573507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.573507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:19.573507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:19.573507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.573507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:19.573507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.573507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:19.573507 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:19.573507 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:19.573507 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:19.573765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:19.573765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.573765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.574519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:19.574519 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.574519 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:19.574519 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:19.574519 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.574519 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:19.574519 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.574519 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:19.575492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:19.576358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:19.576358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:19.576358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.576358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:19.576358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:19.576358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.576358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:19.576358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.576358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:19.576583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.576583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:19.576583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:19.576583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:19.576583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.576583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:19.576583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:19.576583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:19.576583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:19.576583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.576583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:19.576967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:19.577315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.577315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:19.577315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.577315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.577987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:19.580250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:19.580803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.580803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:19.580803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.580803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:19.580803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:19.580803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:19.580803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:19.580803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:19.581029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.581029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:19.581029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.581029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:19.581029 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.581029 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:19.581029 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:19.581029 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.581029 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:19.581029 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.581029 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:19.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:19.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:19.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:19.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:19.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:19.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:19.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:19.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:19.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.582955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:19.583306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.583306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:19.584029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:19.584448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:19.584939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:19.584939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:19.584939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:24.058143 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:24.059674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:24.059674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:24.060323 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:24.114547 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.114785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.115424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:24.115424 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.115424 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:24.115424 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:24.115424 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.115424 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:24.115424 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.115424 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:24.116142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.116142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:24.116142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:24.116142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:24.116142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.116142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:24.116142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:24.116142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.116142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:24.116142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:24.116142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:24.116142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.116835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:24.117570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:24.118280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.118671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:24.118671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:24.118671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.118671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:24.118671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.118671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:24.553621 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:24.555329 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:24.555425 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:24.557453 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.557581 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.557715 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.557839 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.557941 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce8f4a32-94cc-4682-94cb-97f17b86edff, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.558029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.558119 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.558230 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.558344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.558439 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.558522 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f090e551-026c-4a92-b933-a403a0e0e5aa, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.558616 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.558736 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-8838ae75-8094-46bb-aad0-ea97037039d3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.558837 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-7d6e25ee-a0a4-4b25-af85-3f78fd91d8c8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.558926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.559072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.559532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:24.559532 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.559532 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:24.559532 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:24.559532 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.559532 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:24.559532 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.559532 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:24.560142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.560142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:24.560142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:24.560142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:24.560142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.560142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:24.560142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:24.560142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.560142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:24.560142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:24.560142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:24.560142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.560547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:24.561110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:24.561592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.562023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:24.562023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:24.562023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.562023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:24.562023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.562023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:24.566011 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:24.566341 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:24.566418 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:24.566524 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:54:24.566874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.566994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.567421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:24.567421 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.567421 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:24.567421 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:24.567421 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.567421 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:24.567421 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.567421 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:24.568213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:24.568533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:24.568533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:24.568533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.568533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:24.568533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:24.568533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.568533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:24.568533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.568533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:24.569020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.569020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:24.569020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:24.569020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:24.569020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.569020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:24.569020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:24.569020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:54:24.569020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:24.569020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:24.569020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.569020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:24.569450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:24.569878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.569878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:24.569878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.569878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:24.570022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:24.570395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:24.570732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.570732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:24.570732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:24.570732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.570732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:24.570732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.570732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:24.572235 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:24.572558 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:24.572636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:24.572760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:24.573021 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.573215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.573613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:24.573613 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.573613 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:24.573613 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:24.573613 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.573613 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:24.573613 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.573613 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:24.574034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:24.574299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:24.574299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:24.574299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.574299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:24.574299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:24.574299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.574299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:24.574299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.574299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:24.574806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.574806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:24.574806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:24.574806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:24.574806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.574806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:24.574806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:24.574806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:24.574806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:24.574806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.574806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:24.575072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:24.575381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:24.575694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:24.576034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:24.576034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:24.576034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:24.576034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:24.576034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.576034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:24.576034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:24.576034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.576034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:24.576034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.576034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:24.578677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:24.578919 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:24.579004 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:24.579145 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:24.579399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.579577 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:24.579892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.580310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:24.580310 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.580310 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:24.580310 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:24.580310 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.580310 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:24.580310 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.580310 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:24.580893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.580893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:24.580893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:24.580893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:24.580893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.580893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:24.580893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:24.580893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.580893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:24.580893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:24.580893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:24.580893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.581174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:24.581427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.581427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:24.581607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:24.581957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:24.582310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:24.582310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:24.582310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:29.059722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:29.060157 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:29.060157 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:29.060825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:29.109875 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.110199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.110968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:29.110968 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.110968 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:29.110968 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:29.110968 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.110968 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:29.110968 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.110968 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:29.111909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.111909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:29.111909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:29.111909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:29.111909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.111909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:29.111909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:29.111909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.111909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:29.111909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:29.111909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:29.111909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.112154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:29.112433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.112433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:29.112940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:29.113216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.113533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:29.113533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.113533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:29.555404 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:29.555859 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:29.556015 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:29.558713 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.559270 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.559499 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.559704 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.559985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.560151 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.560338 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.560554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.560821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.561273 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f090e551-026c-4a92-b933-a403a0e0e5aa, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.561532 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.561836 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-8838ae75-8094-46bb-aad0-ea97037039d3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.562315 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.562475 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-7d6e25ee-a0a4-4b25-af85-3f78fd91d8c8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.563231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-ce8f4a32-94cc-4682-94cb-97f17b86edff, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.563353 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=15): DelStaticRoutesCommand(_result=None, lrouter=neutron-efa896ef-7c2a-4225-b35d-cef6374545d0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.563466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.563972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:29.563972 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.563972 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:29.563972 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:29.563972 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.563972 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:29.563972 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.563972 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:29.564641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.564641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:29.564641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:29.564641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:29.564641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.564641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:29.564641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:29.564641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.564641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:29.564641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:29.564641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:29.564641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.564865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:29.565130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.565130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:29.565561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:29.565864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.566239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:29.566239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.566239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:29.566395 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:29.566632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:29.566727 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:29.566890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:54:29.567239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.567449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.567902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:29.567902 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.567902 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:29.567902 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:29.567902 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.567902 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:29.567902 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.567902 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:29.568585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:29.569036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:29.569036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:29.569036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.569036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:29.569036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:29.569036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.569036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:29.569036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.569036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:29.569283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.569283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:29.569283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:29.569283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:29.569283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.569283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:29.569283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:29.569283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:54:29.569283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:29.569283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:29.569283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.569283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:29.569632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:29.570024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:29.570461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:29.570995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:29.570995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:29.570995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:29.570995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:29.570995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:29.570995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.570995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:29.570995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:29.570995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.570995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:29.570995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.570995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:29.572601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:29.572905 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:29.572969 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:29.573133 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:29.573502 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.573715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.574156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:29.574156 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.574156 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:29.574156 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:29.574156 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.574156 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:29.574156 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.574156 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:29.574791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:29.575127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:29.575127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:29.575127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.575127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:29.575127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:29.575127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.575127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:29.575127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.575127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:29.575464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.575464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:29.575464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:29.575464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:29.575464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.575464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:29.575464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:29.575464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:29.575464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:29.575464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.575464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:29.575706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:29.575947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.575947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:29.575947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.575947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.576252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:29.576548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:29.576939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.576939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:29.576939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.576939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:29.578954 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:29.579147 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:29.579260 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:29.579426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:29.579693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.579948 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:29.580317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.580820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:29.580820 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.580820 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:29.580820 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:29.580820 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.580820 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:29.580820 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.580820 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:29.581488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.581488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:29.581488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:29.581488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:29.581488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.581488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:29.581488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:29.581488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.581488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:29.581488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:29.581488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:29.581488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.581711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:29.581935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.581935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:29.582365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:29.582675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:29.583027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:29.583027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:29.583027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:34.061608 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:34.062963 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:34.062963 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:34.062963 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:34.107255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.107493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.108063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:34.108063 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.108063 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:34.108063 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:34.108063 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.108063 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:34.108063 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.108063 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:34.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:34.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:34.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:34.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:34.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:34.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:34.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:34.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:34.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.109370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:34.109774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.109774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:34.110234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:34.110574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.110926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:34.110926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.110926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:34.555811 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:34.556074 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:34.556168 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:34.559284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.559477 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.559589 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.559684 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.559810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.559922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.560014 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.560118 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.560376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.560503 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f090e551-026c-4a92-b933-a403a0e0e5aa, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.561337 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.561477 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-8838ae75-8094-46bb-aad0-ea97037039d3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.561595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.561694 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-7d6e25ee-a0a4-4b25-af85-3f78fd91d8c8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.561815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-efa896ef-7c2a-4225-b35d-cef6374545d0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.561929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:34.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:34.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:34.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:34.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.562449 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:34.563087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.563087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:34.563087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:34.563087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:34.563087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.563087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:34.563087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:34.563087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.563087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:34.563087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:34.563087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:34.563087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.563505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:34.563838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:34.564390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.565033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:34.565033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:34.565033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.565033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:34.565033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.565033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:34.566744 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:34.566987 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:34.567097 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:34.567180 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:54:34.567506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.567658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.568088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:34.568088 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.568088 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:34.568088 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:34.568088 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.568088 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:34.568088 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.568088 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:34.568863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:34.569500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:34.569500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:34.569500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.569500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:34.569500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:34.569500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.569500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:34.569500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.569500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:34.569812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.569812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:34.569812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:34.569812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:34.569812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.569812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:34.569812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:34.569812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:54:34.569812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:34.569812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:34.569812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.569812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:34.570139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:34.570523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.570523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:34.570523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.570523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:34.570851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:34.571173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:34.571544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.571544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:34.571544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:34.571544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.571544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:34.571544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.571544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:34.572864 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:34.573113 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:34.573280 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:34.573409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:34.573963 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.574195 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.574585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:34.574585 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.574585 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:34.574585 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:34.574585 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.574585 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:34.574585 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.574585 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:34.575300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:34.575696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:34.575696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:34.575696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.575696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:34.575696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:34.575696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.575696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:34.575696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.575696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:34.576536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.576536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:34.576536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:34.576536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:34.576536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.576536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:34.576536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:34.576536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:34.576536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:34.576536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.576536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:34.576893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:34.577287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.577287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:34.577287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.577287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.577404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:34.577710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:34.578097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.578097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:34.578097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.578097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:34.579272 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:34.579444 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:34.579529 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:34.579660 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:34.579957 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.580299 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:34.580633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.581005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:34.581005 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.581005 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:34.581005 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:34.581005 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.581005 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:34.581005 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.581005 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:34.581726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.581726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:34.581726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:34.581726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:34.581726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.581726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:34.581726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:34.581726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.581726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:34.581726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:34.581726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:34.581726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.582041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:34.582443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:34.582924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:34.583346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:34.583346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:34.583346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:34.583346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:34.583346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:34.583346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:39.062121 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:39.062683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:39.062683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:39.063176 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:39.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.107552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.108094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:39.108094 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.108094 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:39.108094 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:39.108094 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.108094 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:39.108094 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.108094 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:39.108814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.108814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:39.108814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:39.108814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:39.108814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.108814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:39.108814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:39.108814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.108814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:39.108814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:39.108814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:39.108814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.109060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:39.109309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.109309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:39.109552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:39.109971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.110304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:39.110304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.110304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:39.557697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:39.557697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:39.557697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:39.559432 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.559703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.560058 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.560344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.560522 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.560690 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.560887 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.561123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.562032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.562285 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.562558 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-8838ae75-8094-46bb-aad0-ea97037039d3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.562766 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.563023 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-efa896ef-7c2a-4225-b35d-cef6374545d0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.563321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.564438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:39.564438 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.564438 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:39.564438 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:39.564438 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.564438 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:39.564438 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.564438 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:39.567029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.567029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:39.567029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:39.567029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:39.567029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.567029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:39.567029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:39.567029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.567029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:39.567029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:39.567029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:39.567029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.567351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:39.567603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.567603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:39.568565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:39.568935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.569409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:39.569409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.569409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:39.569409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:39.569409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:39.569409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:39.569409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:54:39.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.570450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.570630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:39.570630 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.570630 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:39.570630 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:39.570630 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.570630 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:39.570630 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.570630 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:39.571807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:39.572109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:39.572109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:39.572109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.572109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:39.572109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:39.572109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.572109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:39.572109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.572109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:39.572810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.572810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:39.572810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:39.572810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:39.572810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.572810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:39.572810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:39.572810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:54:39.572810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:39.572810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:39.572810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.572810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:39.573048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:39.573313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.573313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:39.573313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.573313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:39.574198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:39.574464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:39.574780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.574780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:39.574780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:39.574780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.574780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:39.574780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.574780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:39.575041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:39.575402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:39.575533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:39.575683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:39.576441 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.576441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.577544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:39.577544 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.577544 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:39.577544 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:39.577544 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.577544 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:39.577544 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.577544 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:39.578183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:39.578473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:39.578473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:39.578473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.578473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:39.578473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:39.578473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.578473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:39.578473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.578473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:39.579201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.579201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:39.579201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:39.579201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:39.579201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.579201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:39.579201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:39.579201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:39.579201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:39.579201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.579201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:39.579419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:39.579667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.579667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:39.579667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.579667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.581011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:39.581374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:39.581823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.581823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:39.581823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.581823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:39.581823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:39.581823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:39.581823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:39.581992 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:39.583243 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.583243 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:39.583243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.583798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:39.583798 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.583798 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:39.583798 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:39.583798 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.583798 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:39.583798 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.583798 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:39.584475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.584475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:39.584475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:39.584475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:39.584475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.584475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:39.584475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:39.584475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.584475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:39.584475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:39.584475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:39.584475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.584742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:39.584979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.584979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:39.585496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:39.585946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:39.586568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:39.586568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:39.586568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:44.064857 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:44.065471 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:44.065471 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:44.066003 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:44.119975 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.119975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.120652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:44.120652 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.120652 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:44.120652 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:44.120652 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.120652 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:44.120652 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.120652 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:44.121177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.121177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:44.121177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:44.121177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:44.121177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.121177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:44.121177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:44.121177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.121177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:44.121177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:44.121177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:44.121177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.121584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:44.121976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:44.122547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.123183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:44.123183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:44.123183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.123183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:44.123183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.123183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:44.559023 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:44.559023 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:44.559023 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:44.561673 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.562095 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.562207 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.562378 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.562527 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.562683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.562896 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.562989 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.563421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.563421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.563534 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-8838ae75-8094-46bb-aad0-ea97037039d3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.564016 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.564016 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-efa896ef-7c2a-4225-b35d-cef6374545d0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.564016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.564692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:44.564692 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.564692 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:44.564692 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:44.564692 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.564692 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:44.564692 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.564692 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:44.566486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.566486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:44.566486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:44.566486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:44.566486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.566486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:44.566486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:44.566486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.566486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:44.566486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:44.566486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:44.566486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.567184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:44.567646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:44.568559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.569088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:44.569088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:44.569088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.569088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:44.569088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.569088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:44.572777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:44.573209 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:44.573376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:44.573424 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:54:44.575486 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:44.576955 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.577455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.577688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:44.577688 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.577688 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:44.577688 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:44.577688 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.577688 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:44.577688 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.577688 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:44.579090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:44.579908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:44.580254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:44.580517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:44.580517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:44.580517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:44.580517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:44.580517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:44.580517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.580517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:44.580517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:44.580517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.580517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:44.580517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.580517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:44.580957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:44.581306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:44.581919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.581919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:44.581919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:44.581919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.581919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:44.581919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.581919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:44.581919 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:44.581919 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:44.581919 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:44.584101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:44.585294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.585294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.585749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:44.585749 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.585749 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:44.585749 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:44.585749 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.585749 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:44.585749 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.585749 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:44.586652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:44.587175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:44.587175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:44.587175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.587175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:44.587175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:44.587175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.587175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:44.587175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.587175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:44.587547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.587547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:44.587547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:44.587547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:44.587547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.587547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:44.587547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:44.587547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:44.587547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:44.587547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.587547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:44.587943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:44.588370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.588370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:44.588370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.588370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.588699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:44.589520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:44.590092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.590092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:44.590092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.590092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:44.590092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:44.590092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:44.590092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:44.590092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.590391 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:44.590391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.590560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:44.590560 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.590560 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:44.590560 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:44.590560 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.590560 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:44.590560 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.590560 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:44.591406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.591406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:44.591406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:44.591406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:44.591406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.591406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:44.591406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:44.591406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.591406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:44.591406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:44.591406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:44.591406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:44.592547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:44.593155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:44.594289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:44.594289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:44.594289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:44.594289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:44.594289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:44.594289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:49.065835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:49.065835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:49.065835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:49.066551 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:49.125269 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.065 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:49.131285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:49.131650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:49.132292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:49.132763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:49.132763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:49.132763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.132763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:49.132763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:49.132763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.132763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:49.132763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.132763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:49.558840 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:49.561372 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:49.561466 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:49.563249 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.563375 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.563491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.563593 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.563690 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.563790 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.563878 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.563997 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.564093 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.564204 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.564317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-8838ae75-8094-46bb-aad0-ea97037039d3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.564410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.564498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-efa896ef-7c2a-4225-b35d-cef6374545d0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.564580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.564671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.565104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:49.565104 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.565104 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:49.565104 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:49.565104 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.565104 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:49.565104 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.565104 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:49.567204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.567204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:49.567204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:49.567204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:49.567204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.567204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:49.567204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:49.567204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.567204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:49.567204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:49.567204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:49.567204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.567681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:49.568142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:49.568735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.569244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:49.569244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:49.569244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.569244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:49.569244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.569244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:49.573176 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:49.573696 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:49.573788 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:49.573876 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:54:49.574202 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.574380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.574766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:49.574766 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.574766 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:49.574766 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:49.574766 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.574766 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:49.574766 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.574766 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:49.575445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:49.577995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:49.578520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.578773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:49.579325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:49.580153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:49.580491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.580491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:49.580491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:49.580491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.580491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:49.580491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.580491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:49.581351 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:49.581706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:49.581809 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:49.581960 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:49.582477 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.582676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.584062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:49.584062 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.584062 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:49.584062 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:49.584062 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.584062 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:49.584062 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.584062 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:49.584815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:49.585213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:49.585213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:49.585213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.585213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:49.585213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:49.585213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.585213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:49.585213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.585213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:49.585559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.585559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:49.585559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:49.585559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:49.585559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.585559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:49.585559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:49.585559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:49.585559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:49.585559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.585559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:49.586250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:49.586472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:49.586715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:49.587390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:49.587390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:49.587390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:49.587390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:49.587390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.587390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:49.587390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:49.587390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.587390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:49.587390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.587390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:49.589065 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:49.589782 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:49.589782 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:49.589782 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:49.590046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.590393 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:49.590829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.591457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:49.591457 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.591457 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:49.591457 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:49.591457 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.591457 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:49.591457 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.591457 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:49.592359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.592359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:49.592359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:49.592359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:49.592359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.592359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:49.592359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:49.592359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.592359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:49.592359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:49.592359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:49.592359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.592754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:49.593149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.593149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:49.593454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:49.593939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:49.594553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:49.594553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:49.594553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:54.065811 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:54.065811 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:54.066548 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:54.066692 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:54.138317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.138544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.139700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:54.139700 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.139700 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:54.139700 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:54.139700 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.139700 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:54.139700 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.139700 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:54.140942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.074 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.140942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:54.140942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:54.140942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:54.140942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.140942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:54.140942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:54.140942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.140942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:54.140942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:54.140942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:54.140942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.141429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:54.142032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.142032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:54.142799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:54.143401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.144170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:54.144170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.144170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:54.562913 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:54.563448 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:54.563448 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:54.565062 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.565204 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.565344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.565504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.565752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.565824 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.565926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.566078 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.566237 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.566397 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8838ae75-8094-46bb-aad0-ea97037039d3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.566565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.566729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-efa896ef-7c2a-4225-b35d-cef6374545d0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.566974 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.567043 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.567158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.567718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:54.567718 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.567718 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:54.567718 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:54.567718 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.567718 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:54.567718 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.567718 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:54.569207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.569734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:54.570183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:54.570707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.571355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:54.571355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:54.571355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.571355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:54.571355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.571355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:54.575179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:54.575179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:54.575179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:54.575179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:54:54.575179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.575179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.575806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:54.575806 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.575806 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:54.575806 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:54.575806 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.575806 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:54.575806 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.575806 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:54.576222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:54.576710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:54.576710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:54.576710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.576710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:54.576710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:54.576710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.576710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:54.576710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.576710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:54.577022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.577022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:54.577022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:54.577022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:54.577022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.577022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:54.577022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:54.577022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:54:54.577022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:54.577022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:54.577022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.577022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:54.577389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:54.577736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.577736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:54.577736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.577736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:54.578051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:54.578558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:54.579084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.579084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:54.579084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:54.579084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.579084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:54.579084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.579084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:54.581775 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:54.582074 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:54.582199 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:54.582374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:54.582712 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.582963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.584993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:54.586085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:54.586498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.587452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:54.587852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:54.588449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:54.589150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:54.589150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.589150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:54.589150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:54.589150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.589150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:54.589150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.589150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:54.589608 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:54.589954 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:54.590118 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:54.590376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:54.590636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.590882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:54.591307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.591827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:54.591827 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.591827 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:54.591827 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:54.591827 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.591827 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:54.591827 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.591827 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:54.592649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.592649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:54.592649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:54.592649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:54.592649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.592649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:54.592649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:54.592649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.592649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:54.592649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:54.592649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:54.592649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.593050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:54.593474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.593474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:54.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:54.594238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:54.594550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:54.594550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:54.594550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:59.065879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:59.066364 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:59.066459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:59.067337 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:59.123602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.123882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.125181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:59.125181 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.125181 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:59.125181 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:59.125181 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.125181 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:59.125181 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.125181 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:59.126031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.126031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:59.126031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:59.126031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:59.126031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.126031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:59.126031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:59.126031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.126031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:59.126031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:59.126031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:59.126031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.126319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:59.126548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.126548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:59.127781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:59.128162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.128535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:59.128535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.128535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:59.566531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:59.574516 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:59.574639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:59.578532 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:59.579238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.579622 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.579913 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.580817 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.580817 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.580817 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.580817 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.582045 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.583440 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8838ae75-8094-46bb-aad0-ea97037039d3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.583763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.584023 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-efa896ef-7c2a-4225-b35d-cef6374545d0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.584436 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:59.584646 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.585073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.586604 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.586741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.591371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:59.591371 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.591371 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:59.591371 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:59.591371 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.591371 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:59.591371 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.591371 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:59.592101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.017 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.592101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:59.592101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:59.592101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:59.592101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.592101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:59.592101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:59.592101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.592101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:59.592101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:59.592101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:59.592101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.592557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:59.593230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:59.599397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.600148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:59.600148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:59.600148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.600148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:59.600148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.600148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:59.600148 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:59.600148 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:59.600148 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:54:59.600148 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.600486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.601024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:59.601024 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.601024 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:59.601024 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:59.601024 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.601024 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:59.601024 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.601024 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:59.602996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:59.604084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:59.604976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:59.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:59.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:59.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:59.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:59.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:59.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:59.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:59.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:59.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:59.606049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:59.606972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:59.608276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.608276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:59.608276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:59.608276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.608276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:59.608276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.608276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:59.608276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:59.608276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:59.608276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.608951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.609817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:54:59.610910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:54:59.610910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.610910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:54:59.610910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.610910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:54:59.610910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.610910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:59.610910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:59.610910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:59.610910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.610910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:59.610910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:59.611252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:59.611576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:59.611576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:59.611576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.611576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:59.611576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:59.611576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.611576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:59.611576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.611576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:59.611576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.611576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:59.611576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.611867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.614121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:54:59.614693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:54:59.614693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:54:59.614693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:54:59.614693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.614693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:54:59.614693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:59.614914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:54:59.615399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.615744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:54:59.616094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:54:59.616625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:54:59.616625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:54:59.616625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:54:59.616625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:54:59.616625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:54:59.616625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:54:59.616625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:54:59.616625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:54:59.616625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:54:59.616625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:04.067214 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:04.067813 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:04.067813 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:04.069267 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:04.123313 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.123596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.124270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:04.124270 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.124270 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:04.124270 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:04.124270 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.124270 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:04.124270 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.124270 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:04.125376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.125376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:04.125376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:04.125376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:04.125376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.125376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:04.125376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:04.125376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.125376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:04.125376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:04.125376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:04.125376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.125741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:04.126113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.126113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:04.127033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:04.127585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.128212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:04.128212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.128212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:04.575518 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:04.575876 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:04.576063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:04.579630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.579808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.579921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e6faac3c-5ffe-441f-ac24-b6818a34bad8, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.580040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.580146 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.580274 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.580375 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.580477 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8838ae75-8094-46bb-aad0-ea97037039d3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.580604 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.580707 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-efa896ef-7c2a-4225-b35d-cef6374545d0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.580804 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.580905 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.581061 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.581213 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.581334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.582057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:04.582057 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.582057 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:04.582057 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:04.582057 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.582057 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:04.582057 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.582057 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:04.583104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.583104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:04.583104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:04.583104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:04.583104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.583104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:04.583104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:04.583104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.583104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:04.583104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:04.583104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:04.583104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.583545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:04.583810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.583810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:04.588211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:04.588659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.589094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:04.589094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.589094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:04.593410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:04.593739 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:04.596409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:04.596689 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:55:04.597380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.597628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.598255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:04.598255 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.598255 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:04.598255 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:04.598255 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.598255 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:04.598255 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.598255 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:04.599376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:04.599796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:04.599796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:04.599796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.599796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:04.599796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:04.599796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.599796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:04.599796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.599796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:04.600691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.600691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:04.600691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:04.600691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:04.600691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.600691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:04.600691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:04.600691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:55:04.600691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:04.600691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:04.600691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.600691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:04.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:04.601300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.601300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:04.601300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.601300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:04.602216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:04.602556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:04.602876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.602876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:04.602876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:04.602876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.602876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:04.602876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.602876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:04.603783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:04.605537 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:04.605812 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:04.606008 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:04.608259 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.608525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.609256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:04.609256 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.609256 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:04.609256 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:04.609256 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.609256 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:04.609256 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.609256 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:04.610236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:04.610714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:04.610714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:04.610714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.610714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:04.610714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:04.610714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.610714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:04.610714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.610714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:04.614639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:04.615564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.615564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:04.615564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:04.615564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:04.615564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.615564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:04.615564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:04.615564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:04.615564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:04.615564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.615564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:04.615875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:04.616435 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.616435 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:04.616435 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.616435 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.617146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:04.617523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:04.617924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.617924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:04.617924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.617924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:04.620956 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:04.621128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:04.621363 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:04.624914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.625249 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:04.625704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.626423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:04.626423 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.626423 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:04.626423 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:04.626423 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.626423 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:04.626423 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.626423 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:04.627794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.627794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:04.627794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:04.627794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:04.627794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.627794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:04.627794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:04.627794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.627794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:04.627794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:04.627794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:04.627794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.628112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:04.628392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.628392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:04.629262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:04.629752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:04.630333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:04.630333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:04.630333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:09.067666 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:09.068455 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:09.068559 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:09.069197 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:09.125651 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.125916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.126649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:09.126649 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.126649 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:09.126649 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:09.126649 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.126649 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:09.126649 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.126649 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:09.128624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.128624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:09.128624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:09.128624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:09.128624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.128624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:09.128624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:09.128624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.128624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:09.128624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:09.128624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:09.128624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:09.129324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.129324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:09.129906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:09.130440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.130900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:09.130900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.130900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:09.576574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:09.576915 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:09.577072 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:09.578655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.578871 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.579036 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.582046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.582046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.582046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.582046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8838ae75-8094-46bb-aad0-ea97037039d3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.582046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.582263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-efa896ef-7c2a-4225-b35d-cef6374545d0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.582263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.582263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.582263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.582263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.582263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:09.582408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:09.582822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.583072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:09.583072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:09.583072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.583072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:09.583072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.583072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:09.583821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:09.584115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.584449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:09.584449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.584449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:09.593902 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:09.594712 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:09.594861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:09.595034 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:55:09.596271 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.597134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.597134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:09.597134 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.597134 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:09.597134 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:09.597134 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.597134 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:09.597134 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.597134 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:09.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:09.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:09.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:09.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:09.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:09.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:09.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:09.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:09.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:09.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:09.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:09.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:09.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:55:09.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:09.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:09.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:09.600024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:09.600484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.600484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:09.600484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.600484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:09.603253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:09.603802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:09.604337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.604337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:09.604337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:09.604337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.604337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:09.604337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.604337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:09.606200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:09.606200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:09.606200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:09.606200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:09.607556 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.607795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.612076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:09.612753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:09.613211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.613587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:09.613905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:09.614333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:09.614819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:09.614819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.614819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:09.614819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:09.614819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.614819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:09.614819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.614819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:09.621158 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:09.621285 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:09.621557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:09.621692 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:09.622436 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.622699 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:09.623571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.623852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:09.623852 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.623852 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:09.623852 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:09.623852 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.623852 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:09.623852 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.623852 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:09.624707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.624707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:09.624707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:09.624707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:09.624707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.624707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:09.624707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:09.624707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.624707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:09.624707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:09.624707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:09.624707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.624977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:09.625230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.625230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:09.625618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:09.625925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:09.626283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:09.626283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:09.626283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:14.068749 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:14.072517 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:14.072517 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:14.072517 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:14.145739 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.146260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.147179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:14.147179 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.147179 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:14.147179 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:14.147179 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.147179 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:14.147179 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.147179 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:14.150409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.077 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.150409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:14.150409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:14.150409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:14.150409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.150409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:14.150409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:14.150409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.150409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:14.150409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:14.150409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:14.150409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.151461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:14.154354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.154354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:14.156366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:14.157713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.158136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:14.158136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.158136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:14.578066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:14.579447 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:14.579699 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:14.582110 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.583042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-89738a37-54e7-4b8b-9e98-a3da98970561, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.583042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.583042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.583042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.583474 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.583474 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.583474 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.583616 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.583760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.584019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.589296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.589296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:14.589296 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.589296 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:14.589296 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:14.589296 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.589296 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:14.589296 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.589296 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:14.591054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.591054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:14.591054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:14.591054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:14.591054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.591054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:14.591054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:14.591054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.591054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:14.591054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:14.591054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:14.591054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.592169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:14.593063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.593063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:14.596065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:14.596530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.596853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:14.596853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.596853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:14.596853 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:14.597205 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:14.597205 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:14.597330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:55:14.598017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.598332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.600686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:14.600686 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.600686 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:14.600686 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:14.600686 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.600686 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:14.600686 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.600686 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:14.601391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:14.601797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:14.601797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:14.601797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.601797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:14.601797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:14.601797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.601797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:14.601797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.601797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:14.602246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.602246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:14.602246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:14.602246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:14.602246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.602246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:14.602246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:14.602246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:55:14.602246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:14.602246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:14.602246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.602246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:14.603226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:14.603603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.603603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:14.603603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.603603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:14.607417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:14.607862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:14.608360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.608360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:14.608360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:14.608360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.608360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:14.608360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.608360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:14.611321 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:14.611321 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:14.611321 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:14.611321 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:14.612595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.612910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.615755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:14.615755 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.615755 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:14.615755 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:14.615755 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.615755 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:14.615755 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.615755 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:14.616976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:14.617707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:14.618363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:14.618950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:14.618950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:14.618950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:14.618950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:14.618950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.618950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:14.618950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:14.618950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.618950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:14.618950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.618950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.631606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:14.633124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:14.633883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.633883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:14.633883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.633883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:14.633883 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:14.639114 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:14.639114 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:14.639114 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:14.639114 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.639372 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:14.639662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:14.642909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:14.643294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.643543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:14.643543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:14.643543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.643543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:14.643543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.643543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:14.644153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:14.644461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:14.644782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:14.644782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:14.644782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:19.070362 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:19.070936 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:19.070936 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:19.072244 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:19.074350 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=240 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 00:55:19.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.133731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.134679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:19.134679 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.134679 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:19.134679 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:19.134679 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.134679 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:19.134679 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.134679 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:19.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.064 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:19.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:19.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:19.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:19.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:19.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:19.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:19.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:19.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.135839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:19.136268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:19.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.137366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:19.137366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:19.137366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.137366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:19.137366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.137366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:19.580727 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:19.582359 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:19.582359 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:19.583229 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.585133 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.585133 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.585133 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.585133 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.586163 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.586527 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.586723 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.586988 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.587494 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.587494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.589066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:19.589066 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.589066 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:19.589066 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:19.589066 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.589066 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:19.589066 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.589066 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:19.590663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.590663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:19.590663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:19.590663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:19.590663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.590663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:19.590663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:19.590663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.590663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:19.590663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:19.590663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:19.590663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.590989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:19.591264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.591264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:19.591481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:19.591760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.592086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:19.592086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.592086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:19.597107 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:19.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:19.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:19.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:55:19.599709 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.601681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.602748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:19.602748 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.602748 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:19.602748 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:19.602748 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.602748 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:19.602748 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.602748 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:19.603743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:19.604308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:19.604308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:19.604308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.604308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:19.604308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:19.604308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.604308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:19.604308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.604308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:19.605307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.605307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:19.605307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:19.605307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:19.605307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.605307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:19.605307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:19.605307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:55:19.605307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:19.605307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:19.605307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.605307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:19.605636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:19.606088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:19.606604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:19.607725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:19.607725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:19.607725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:19.607725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:19.607725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:19.607725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.607725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:19.607725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:19.607725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.607725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:19.607725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.607725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:19.611052 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:19.611052 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:19.611675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:19.611675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:19.615314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.615779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:19.616351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.616351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:19.617588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.617588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:19.617588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:19.617588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:19.617588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.617588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:19.617588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:19.617588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:19.617588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:19.617588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.617588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:19.618151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:19.618544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.618544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:19.618544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.618544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.620736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:19.622562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:19.624018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.624018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:19.624018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.624018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:19.635076 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:19.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:19.636595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:19.636595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:19.636892 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.637174 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:19.637530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:19.640128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:19.640507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.640791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:19.641082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:19.641460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:19.641460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:19.641460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:19.641460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:19.641460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:19.641460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:19.641460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:19.641460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:19.641460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:19.641460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:24.070910 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:24.071529 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:24.071529 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:24.072240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:24.118415 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.118415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.119238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:24.119238 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.119238 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:24.119238 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:24.119238 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.119238 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:24.119238 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.119238 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:24.119930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.119930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:24.119930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:24.119930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:24.119930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.119930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:24.119930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:24.119930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.119930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:24.119930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:24.119930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:24.119930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.120441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:24.121065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.121065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:24.121146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:24.121742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.122781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:24.122781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.122781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:24.581708 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:24.583533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:24.583655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:24.586210 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.587120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.590622 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.590768 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.591011 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.592230 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.592230 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.592230 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.592230 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.592230 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.592230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.592872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:24.592872 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.592872 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:24.592872 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:24.592872 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.592872 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:24.592872 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.592872 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:24.594455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.594455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:24.594455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:24.594455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:24.594455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.594455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:24.594455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:24.594455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.594455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:24.594455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:24.594455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:24.594455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.594857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:24.595284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.595284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:24.598886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:24.599456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.599923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:24.599923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.599923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:24.600804 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:24.602702 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:24.603313 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:24.604131 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:55:24.607591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.608298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.608603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:24.608603 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.608603 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:24.608603 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:24.608603 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.608603 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:24.608603 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.608603 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:24.609739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:24.610253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:24.610253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:24.610253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.610253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:24.610253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:24.610253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.610253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:24.610253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.610253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:24.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:24.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:24.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:24.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:24.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:24.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:55:24.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:24.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:24.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:24.613860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:24.614250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.614250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:24.614250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.614250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:24.614250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:24.615094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:24.615830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:24.616340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.616340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:24.616340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:24.616340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.616340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:24.616340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.616340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:24.616340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:24.616340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:24.616340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:24.616670 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.616670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.617343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:24.617343 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.617343 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:24.617343 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:24.617343 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.617343 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:24.617343 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.617343 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:24.621660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:24.621976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:24.621976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:24.621976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.621976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:24.621976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:24.621976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.621976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:24.621976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.621976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:24.622544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.622544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:24.622544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:24.622544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:24.622544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.622544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:24.622544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:24.622544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:24.622544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:24.622544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.622544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:24.623220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:24.623595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:24.624105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:24.624473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:24.624473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:24.624473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:24.624473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:24.624473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.624473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:24.624473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:24.624473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.624473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:24.624473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.624473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:24.635689 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:24.636300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:24.636473 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:24.636668 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:24.637108 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.637407 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:24.637950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.640684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:24.640684 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.640684 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:24.640684 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:24.640684 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.640684 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:24.640684 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.640684 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:24.642033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.642033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:24.642033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:24.642033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:24.642033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.642033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:24.642033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:24.642033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.642033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:24.642033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:24.642033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:24.642033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.642512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:24.642869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.642869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:24.644381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:24.644854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:24.645448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:24.645448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:24.645448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:29.071311 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:29.071765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:29.073340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:29.073340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:29.138669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.138863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.139501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:29.139501 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.139501 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:29.139501 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:29.139501 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.139501 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:29.139501 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.139501 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:29.140874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.068 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.140874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:29.140874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:29.140874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:29.140874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.140874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:29.140874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:29.140874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.140874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:29.140874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:29.140874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:29.140874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.141527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:29.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:29.144455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.144834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:29.144834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:29.144834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.144834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:29.144834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.144834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:29.583641 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:29.585296 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:29.585296 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:29.586077 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.588891 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.589660 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.590114 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.590552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.590983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.592500 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.592500 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.592500 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.592500 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.592500 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.592500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.592977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:29.592977 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.592977 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:29.592977 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:29.592977 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.592977 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:29.592977 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.592977 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:29.595080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.595080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:29.595080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:29.595080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:29.595080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.595080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:29.595080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:29.595080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.595080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:29.595080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:29.595080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:29.595080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.595505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:29.596262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:29.596496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.596869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:29.596869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:29.596869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.596869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:29.596869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.596869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:29.603347 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:29.604132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:29.604132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:29.604132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:55:29.605168 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.605168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.605401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:29.605401 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.605401 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:29.605401 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:29.605401 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.605401 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:29.605401 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.605401 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:29.606296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:29.606720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:29.606720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:29.606720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.606720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:29.606720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:29.606720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.606720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:29.606720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.606720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:29.607942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.607942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:29.607942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:29.607942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:29.607942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.607942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:29.607942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:29.607942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:55:29.607942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:29.607942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:29.607942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.607942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:29.608340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:29.608740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.608740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:29.608740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.608740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:29.610317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:29.610947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:29.611480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.611480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:29.611480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:29.611480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.611480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:29.611480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.611480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:29.615456 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:29.616413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:29.616413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:29.616413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:29.617047 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.618110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.618859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:29.618859 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.618859 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:29.618859 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:29.618859 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.618859 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:29.618859 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.618859 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:29.620079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:29.621015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:29.621015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:29.621015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.621015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:29.621015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:29.621015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.621015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:29.621015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.621015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:29.623271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.623271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:29.623271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:29.623271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:29.623271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.623271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:29.623271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:29.623271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:29.623271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:29.623271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.623271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:29.623867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:29.624242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.624242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:29.624242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.624242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.624767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:29.626405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:29.626910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.626910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:29.626910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.626910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:29.637272 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:29.637539 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:29.637692 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:29.637783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:29.638042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.638348 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:29.638690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.639263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:29.639263 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.639263 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:29.639263 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:29.639263 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.639263 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:29.639263 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.639263 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:29.640214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.640214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:29.640214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:29.640214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:29.640214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.640214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:29.640214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:29.640214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.640214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:29.640214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:29.640214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:29.640214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.640606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:29.641760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:29.642048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:29.642447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:29.642447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:29.642447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:29.642447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:29.642447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:29.642447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:34.071840 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:34.072711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:34.072905 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:34.073881 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:34.150287 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.150930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.151919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:34.151919 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.151919 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:34.151919 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:34.151919 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.151919 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:34.151919 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.151919 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:34.153025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.079 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.153025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:34.153025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:34.153025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:34.153025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.153025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:34.153025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:34.153025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.153025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:34.153025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:34.153025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:34.153025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.153454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:34.154415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:34.155017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.155527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:34.155527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:34.155527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.155527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:34.155527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.155527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:34.584216 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:34.584954 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:34.585398 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:34.587680 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.590118 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.590118 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.590118 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.590118 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.590118 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.590366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.590366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.590366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.590366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.590366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.590465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.590465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.590465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:34.590465 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.590465 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:34.590465 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:34.590465 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.590465 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:34.590465 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.590465 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:34.591294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.591294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:34.591294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:34.591294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:34.591294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.591294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:34.591294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:34.591294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.591294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:34.591294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:34.591294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:34.591294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.591548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:34.591761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.591761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:34.592478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:34.593009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.593499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:34.593499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.593499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:34.603984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:34.604667 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:34.605736 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:34.606280 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:55:34.607341 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.607341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.607488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:34.607488 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.607488 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:34.607488 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:34.607488 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.607488 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:34.607488 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.607488 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:34.608350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:34.608815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:34.608815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:34.608815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.608815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:34.608815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:34.608815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.608815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:34.608815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.608815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:34.610965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.610965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:34.610965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:34.610965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:34.610965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.610965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:34.610965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:34.610965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:55:34.610965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:34.610965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:34.610965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.610965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:34.611437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:34.611665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.611665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:34.611665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.611665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:34.612778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:34.613108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:34.613470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.613470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:34.613470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:34.613470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.613470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:34.613470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.613470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:34.615981 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:34.616382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:34.616575 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:34.616754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:34.617681 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.617681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.619239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:34.619239 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.619239 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:34.619239 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:34.619239 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.619239 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:34.619239 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.619239 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:34.620488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:34.620777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:34.620777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:34.620777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.620777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:34.620777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:34.620777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.620777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:34.620777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.620777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:34.621636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.621636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:34.621636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:34.621636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:34.621636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.621636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:34.621636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:34.621636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:34.621636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:34.621636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.621636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:34.621861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:34.622146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.622146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:34.622146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.622146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.624375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:34.624744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:34.625066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.625066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:34.625066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.625066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:34.637744 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:34.638169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:34.638369 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:34.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:34.638907 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.639236 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:34.639550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.640168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:34.640168 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.640168 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:34.640168 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:34.640168 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.640168 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:34.640168 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.640168 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:34.641498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.641498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:34.641498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:34.641498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:34.641498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.641498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:34.641498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:34.641498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.641498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:34.641498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:34.641498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:34.641498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.641792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:34.642068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.642068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:34.642695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:34.643107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:34.643506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:34.643506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:34.643506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:39.072715 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:39.073336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:39.073336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:39.074124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:39.123542 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.123870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.124413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:39.124413 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.124413 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:39.124413 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:39.124413 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.124413 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:39.124413 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.124413 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:39.125561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.125561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:39.125561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:39.125561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:39.125561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.125561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:39.125561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:39.125561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.125561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:39.125561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:39.125561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:39.125561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.125912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:39.126237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.126237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:39.126700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:39.126972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.127306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:39.127306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.127306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:39.584964 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:39.585478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:39.585631 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:39.587153 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.587372 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.587481 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.588263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.588263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.588263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.588263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.588263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.588459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.588596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.588811 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.588943 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.589096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.589796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:39.589796 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.589796 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:39.589796 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:39.589796 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.589796 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:39.589796 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.589796 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:39.592250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.592250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:39.592250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:39.592250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:39.592250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.592250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:39.592250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:39.592250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.592250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:39.592250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:39.592250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:39.592250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.592632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:39.593002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.593002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:39.594059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:39.594561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.595113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:39.595113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.595113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:39.607402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:39.607402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:39.607402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:39.607402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:55:39.607402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.607402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.607784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:39.607784 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.607784 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:39.607784 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:39.607784 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.607784 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:39.607784 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.607784 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:39.608054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:39.608425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:39.608425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:39.608425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.608425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:39.608425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:39.608425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.608425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:39.608425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.608425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:39.608900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.608900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:39.608900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:39.608900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:39.608900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.608900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:39.608900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:39.608900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:55:39.608900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:39.608900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:39.608900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.608900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:39.609272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:39.609647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.609647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:39.609647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.609647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:39.610040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:39.610462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:39.610894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.610894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:39.610894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:39.610894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.610894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:39.610894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.610894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:39.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:39.616900 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:39.617519 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:39.617728 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:39.618228 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.618498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.620746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:39.621357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:39.621738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.622113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:39.622454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.622454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.623527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:39.624022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:39.624560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.624560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:39.624560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.624560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:39.639940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:39.640893 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:39.642284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:39.642553 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:39.643420 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.643711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:39.644833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.644833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:39.644833 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.644833 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:39.644833 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:39.644833 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.644833 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:39.644833 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.644833 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:39.645968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.645968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:39.645968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:39.645968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:39.645968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.645968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:39.645968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:39.645968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.645968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:39.645968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:39.645968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:39.645968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.646368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:39.646614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.646614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:39.647070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:39.647358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:39.647649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:39.647649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:39.647649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:44.073507 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:44.077422 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:44.077520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:44.078100 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:44.137092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.137328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.137859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:44.137859 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.137859 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:44.137859 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:44.137859 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.137859 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:44.137859 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.137859 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:44.138707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.060 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.138707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:44.138707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:44.138707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:44.138707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.138707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:44.138707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:44.138707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.138707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:44.138707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:44.138707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:44.138707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.139057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:44.139382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.139382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:44.139641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:44.140039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.143815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:44.143815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.143815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:44.585474 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:44.586449 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:44.586538 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:44.591491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.591491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.591491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.591491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.591491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.593543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.593543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.593543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.593773 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.593773 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.593773 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.593848 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.593952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.594473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:44.594473 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.594473 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:44.594473 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:44.594473 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.594473 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:44.594473 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.594473 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:44.596467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.596467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:44.596467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:44.596467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:44.596467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.596467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:44.596467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:44.596467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.596467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:44.596467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:44.596467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:44.596467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.596947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:44.597223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:44.597479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.597915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:44.597915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:44.597915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.597915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:44.597915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.597915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:44.605447 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:44.607479 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:44.607609 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:44.607750 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:55:44.608077 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.608371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.612162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:44.612162 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.612162 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:44.612162 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:44.612162 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.612162 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:44.612162 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.612162 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:44.612574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:44.613318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:44.613925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:44.614393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:44.614393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:44.614393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:44.614393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:44.614393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:44.614393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.614393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:44.614393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:44.614393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.614393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:44.614393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.614393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:44.617462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:44.617887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:44.619128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.619128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:44.619128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:44.619128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.619128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:44.619128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.619128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:44.619128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:44.619128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:44.619128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:44.619437 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:44.619916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.620172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.620841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:44.620841 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.620841 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:44.620841 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:44.620841 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.620841 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:44.620841 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.620841 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:44.623603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:44.623961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:44.623961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:44.623961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.623961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:44.623961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:44.623961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.623961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:44.623961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.623961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:44.624621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.624621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:44.624621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:44.624621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:44.624621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.624621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:44.624621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:44.624621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:44.624621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:44.624621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.624621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:44.624954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:44.625301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.625301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:44.625301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.625301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.625664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:44.626079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:44.626463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.626463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:44.626463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.626463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:44.640751 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:44.642403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:44.642403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:44.643500 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:44.644313 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.644565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:44.644934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.651264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:44.651264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.651264 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:44.651264 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:44.651264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.651264 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:44.651264 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.651264 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:44.652064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.652064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:44.652064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:44.652064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:44.652064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.652064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:44.652064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:44.652064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.652064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:44.652064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:44.652064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:44.652064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.652522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:44.652872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:44.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:44.653836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:44.653836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:44.653836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:44.653836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:44.653836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:44.653836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:49.077842 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:49.078348 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:49.078348 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:49.078949 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:49.153671 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.157175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:49.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:49.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:49.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:49.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:49.159635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.080 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.159635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:49.159635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:49.159635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:49.159635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.159635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:49.159635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:49.159635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.159635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:49.159635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:49.159635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:49.159635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.159975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:49.160286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.160286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:49.161883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:49.162272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.162588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:49.162588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.162588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:49.586617 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:49.587030 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:49.587148 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:49.588477 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.588718 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.588897 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.589058 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.589122 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.589293 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.589431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.589559 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.589693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.589831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.589967 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.590129 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.590282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.590873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:49.590873 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.590873 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:49.590873 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:49.590873 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.590873 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:49.590873 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.590873 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:49.591913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.591913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:49.591913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:49.591913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:49.591913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.591913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:49.591913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:49.591913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.591913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:49.591913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:49.591913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:49.591913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.592370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:49.592734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.592734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:49.592914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:49.593396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.593875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:49.593875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.593875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:49.607604 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:49.609703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:49.609851 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:49.610020 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:55:49.610384 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.610572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.611069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:49.611069 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.611069 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:49.611069 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:49.611069 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.611069 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:49.611069 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.611069 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:49.611982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:49.612319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:49.612319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:49.612319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.612319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:49.612319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:49.612319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.612319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:49.612319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.612319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:49.613522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.613522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:49.613522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:49.613522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:49.613522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.613522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:49.613522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:49.613522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:55:49.613522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:49.613522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:49.613522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.613522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:49.613781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:49.614034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.614034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:49.614034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.614034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:49.614811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:49.615130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:49.615507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.615507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:49.615507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:49.615507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.615507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:49.615507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.615507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:49.618891 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:49.619667 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:49.619816 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:49.619982 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:49.620357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.620579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.621058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:49.621058 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.621058 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:49.621058 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:49.621058 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.621058 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:49.621058 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.621058 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:49.621888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:49.622340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:49.622340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:49.622340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.622340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:49.622340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:49.622340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.622340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:49.622340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.622340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:49.622836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.622836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:49.622836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:49.622836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:49.622836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.622836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:49.622836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:49.622836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:49.622836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:49.622836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.622836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:49.623072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:49.623326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.623326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:49.623326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.623326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.624010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:49.624305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:49.624599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.624599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:49.624599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.624599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:49.641374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:49.641977 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:49.642201 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:49.642468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:49.643008 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.643321 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:49.643716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.644566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:49.644566 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.644566 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:49.644566 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:49.644566 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.644566 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:49.644566 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.644566 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:49.645558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.645558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:49.645558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:49.645558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:49.645558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.645558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:49.645558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:49.645558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.645558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:49.645558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:49.645558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:49.645558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.645791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:49.646029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.646029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:49.646553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:49.646920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:49.647266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:49.647266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:49.647266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:54.078472 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:54.079034 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:54.079034 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:54.079588 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:54.117675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.118040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.118628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:54.118628 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.118628 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:54.118628 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:54.118628 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.118628 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:54.118628 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.118628 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:54.119339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.119339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:54.119339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:54.119339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:54.119339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.119339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:54.119339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:54.119339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.119339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:54.119339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:54.119339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:54.119339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.119607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:54.119891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.119891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:54.120059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:54.120395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.120729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:54.120729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.120729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:54.587146 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:54.587588 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:54.587707 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:54.589076 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.589315 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.589454 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.589510 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.589652 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.589797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.589901 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.590017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.590105 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.590255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8131b502-a745-443b-8028-0d658f37a98b, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.590341 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.590451 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.590546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.592465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.592465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:54.592465 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.592465 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:54.592465 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:54.592465 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.592465 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:54.592465 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.592465 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:54.593727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.593727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:54.593727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:54.593727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:54.593727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.593727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:54.593727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:54.593727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.593727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:54.593727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:54.593727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:54.593727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.594180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:54.594586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.594586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:54.595616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:54.596105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.596619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:54.596619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.596619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:54.609637 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:54.610204 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:54.610830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.613247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.613733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:54.614125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:54.614125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.614125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:54.614125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.614125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:54.615201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.615201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:54.615201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:54.615201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:54.615201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.615201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:54.615201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:54.615201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:55:54.615201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:54.615201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:54.615201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.615201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:54.615482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:54.615763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.615763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:54.615763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.615763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:54.616400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:54.616737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:54.617109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.617109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:54.617109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:54.617109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.617109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:54.617109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.617109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:54.619803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:54.620409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:54.620530 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:54.620666 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:54.621136 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.621885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:54.621885 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.621885 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:54.621885 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:54.621885 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.621885 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:54.621885 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.621885 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:54.622739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:54.623066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:54.623066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:54.623066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.623066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:54.623066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:54.623066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.623066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:54.623066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.623066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:54.623629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.623629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:54.623629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:54.623629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:54.623629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.623629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:54.623629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:54.623629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:54.623629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:54.623629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.623629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:54.623978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:54.624283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.624283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:54.624283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.624283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.624697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:54.625134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:54.625505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.625505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:54.625505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.625505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:54.642017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:54.642426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:54.642710 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:54.642710 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:54.643831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.643927 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:54.644282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.644807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:54.644807 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.644807 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:54.644807 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:54.644807 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.644807 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:54.644807 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.644807 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:54.645779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.645779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:54.645779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:54.645779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:54.645779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.645779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:54.645779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:54.645779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.645779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:54.645779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:54.645779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:54.645779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.646300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:54.646715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.646715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:54.646844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:54.647389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:54.648476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:54.648476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:54.648476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:59.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:59.079643 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:59.079643 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:59.080379 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:59.123336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.123532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.124016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:59.124016 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.124016 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:59.124016 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:59.124016 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.124016 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:59.124016 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.124016 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:59.125024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.125024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:59.125024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:59.125024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:59.125024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.125024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:59.125024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:59.125024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.125024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:59.125024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:59.125024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:59.125024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.125857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:59.126606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:59.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.128452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:59.128452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:59.128452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.128452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:59.128452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.128452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:59.587631 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:59.588136 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:59.588206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:59.589167 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.589393 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-1f83a2ae-f62c-4fbd-a0cb-2657320fe2ab, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.589489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.589585 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.589699 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.589820 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.589953 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.590063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.590164 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.590318 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b6a62a0-0a81-41e0-84fd-83d22c3921c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.590439 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.590568 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-cceae05c-b297-4b74-8cf3-1d69fb62fa84, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.590683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.591324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:59.591324 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.591324 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:59.591324 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:59.591324 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.591324 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:59.591324 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.591324 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:59.592068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.592068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:59.592068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:59.592068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:59.592068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.592068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:59.592068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:59.592068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.592068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:59.592068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:59.592068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:59.592068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.592363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:59.592567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.592567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:59.592841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:59.593211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.593552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:59.593552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.593552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:59.610310 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:59.610537 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:59.610617 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:59.610733 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:55:59.611037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.611206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.611700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:59.611700 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.611700 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:59.611700 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:59.611700 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.611700 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:59.611700 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.611700 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:59.612205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:59.612628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:59.612628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:59.612628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.612628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:59.612628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:59.612628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.612628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:59.612628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.612628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:59.612871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.612871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:59.612871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:59.612871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:59.612871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.612871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:59.612871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:59.612871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:55:59.612871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:59.612871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:59.612871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.612871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:59.613231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:59.613589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:59.613974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:59.614509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:59.614509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:59.614509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:59.614509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:59.614509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:59.614509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.614509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:59.614509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:59.614509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.614509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:59.614509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.614509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:59.620473 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:59.620678 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:59.620766 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:59.620871 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:59.621203 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.621470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.622079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:59.622079 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.622079 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:59.622079 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:59.622079 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.622079 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:59.622079 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.622079 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:55:59.622721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:55:59.623079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:59.623079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:55:59.623079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.623079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:55:59.623079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:55:59.623079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.623079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:55:59.623079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.623079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:55:59.623647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.623647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:59.623647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:59.623647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:59.623647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.623647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:59.623647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:59.623647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:59.623647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:59.623647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.623647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:59.623965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:59.624274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.624274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:59.624274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.624274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.624680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:59.625246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:59.625623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.625623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:59.625623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.625623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:55:59.642439 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:55:59.642692 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:55:59.642906 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:55:59.642998 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:55:59.643351 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.643596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:55:59.643944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.644480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:55:59.644480 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.644480 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:55:59.644480 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:55:59.644480 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.644480 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:55:59.644480 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.644480 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:55:59.645349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.645349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:55:59.645349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:59.645349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:55:59.645349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.645349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:59.645349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:59.645349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.645349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:59.645349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:59.645349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:55:59.645349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.645738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:55:59.646062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.646062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:55:59.646389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:55:59.646705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:55:59.647053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:55:59.647053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:55:59.647053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:04.079853 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:04.080829 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:04.080925 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:04.081968 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:04.135469 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.135680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.136139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:04.136139 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.136139 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:04.136139 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:04.136139 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.136139 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:04.136139 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.136139 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:04.137242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.137242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:04.137242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:04.137242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:04.137242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.137242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:04.137242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:04.137242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.137242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:04.137242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:04.137242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:04.137242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.137684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:04.138119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.138119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:04.138560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:04.139059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.139669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:04.139669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.139669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:04.588922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:04.589374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:04.589469 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:04.590441 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.591086 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.591381 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6338c89-ba87-487d-b19d-b524d244ca76, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.591574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.591736 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.591978 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.592197 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.592360 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.592510 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.592674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.594315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:04.594315 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.594315 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:04.594315 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:04.594315 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.594315 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:04.594315 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.594315 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:04.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:04.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:04.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:04.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:04.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:04.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:04.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:04.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:04.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.595971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:04.596248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.596248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:04.597953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:04.598357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.598669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:04.598669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.598669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:04.610677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:04.611775 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:04.611859 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:04.611952 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:56:04.612579 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.612768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:04.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:04.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:04.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:04.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.613511 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:04.615103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:04.615655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:04.615655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:04.615655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.615655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:04.615655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:04.615655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.615655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:04.615655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.615655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:04.616803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.616803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:04.616803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:04.616803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:04.616803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.616803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:04.616803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:04.616803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:56:04.616803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:04.616803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:04.616803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.616803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:04.617282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:04.617647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.617647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:04.617647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.617647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:04.618514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:04.619026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:04.619627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.619627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:04.619627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:04.619627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.619627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:04.619627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.619627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:04.620983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:04.621250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:04.621318 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:04.621424 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:04.621801 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.622080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.623176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:04.623176 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.623176 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:04.623176 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:04.623176 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.623176 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:04.623176 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.623176 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:04.624210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:04.624662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:04.624662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:04.624662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.624662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:04.624662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:04.624662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.624662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:04.624662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.624662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:04.625846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.625846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:04.625846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:04.625846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:04.625846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.625846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:04.625846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:04.625846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:04.625846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:04.625846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.625846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:04.626254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:04.626616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.626616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:04.626616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.626616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.628442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:04.628935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:04.629937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.629937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:04.629937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.629937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:04.645721 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:04.646813 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:04.646926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:04.647125 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:04.647614 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.647870 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:04.648226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.648796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:04.648796 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.648796 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:04.648796 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:04.648796 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.648796 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:04.648796 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.648796 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:04.649511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.649511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:04.649511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:04.649511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:04.649511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.649511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:04.649511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:04.649511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.649511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:04.649511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:04.649511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:04.649511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.649887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:04.650317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:04.650735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:04.651326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:04.651326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:04.651326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:04.651326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:04.651326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:04.651326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:09.080930 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:09.081664 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:09.081793 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:09.082621 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:09.135207 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.135408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.136891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:09.136891 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.136891 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:09.136891 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:09.136891 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.136891 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:09.136891 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.136891 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:09.138265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.138265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:09.138265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:09.138265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:09.138265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.138265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:09.138265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:09.138265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.138265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:09.138265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:09.138265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:09.138265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.138647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:09.138976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.138976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:09.140469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:09.140768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.141236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:09.141236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.141236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:09.589458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:09.590239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:09.590362 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:09.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.593688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.593976 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.593976 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.593976 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.593976 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.593976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.594585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:09.594585 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.594585 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:09.594585 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:09.594585 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.594585 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:09.594585 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.594585 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:09.595283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.595283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:09.595283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:09.595283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:09.595283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.595283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:09.595283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:09.595283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.595283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:09.595283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:09.595283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:09.595283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.595638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:09.595960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.595960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:09.597525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:09.597910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.598384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:09.598384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.598384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:09.611930 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:09.612478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:09.612615 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:09.612759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:56:09.613825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.616770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.617943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:09.617943 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.617943 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:09.617943 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:09.617943 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.617943 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:09.617943 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.617943 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:09.621395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:09.622120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:09.622120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:09.622120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.622120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:09.622120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:09.622120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.622120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:09.622120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.622120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:09.623392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.623392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:09.623392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:09.623392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:09.623392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.623392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:09.623392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:09.623392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:56:09.623392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:09.623392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:09.623392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.623392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:09.623997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:09.624428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.624428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:09.624428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.624428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:09.624428 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:09.627840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:09.629312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:09.629832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.629832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:09.629832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:09.629832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.629832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:09.629832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.629832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:09.629832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:09.629832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:09.631581 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:09.633041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.633041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.633587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:09.633587 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.633587 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:09.633587 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:09.633587 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.633587 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:09.633587 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.633587 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:09.634265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:09.634738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:09.634738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:09.634738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.634738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:09.634738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:09.634738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.634738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:09.634738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.634738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:09.635008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.635008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:09.635008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:09.635008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:09.635008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.635008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:09.635008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:09.635008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:09.635008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:09.635008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.635008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:09.635346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:09.635767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.635767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:09.635767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.635767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.636946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:09.637544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:09.638413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.638413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:09.638413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.638413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:09.646456 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:09.647326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:09.647326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:09.647326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:09.648642 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.649021 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:09.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.650055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:09.650055 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.650055 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:09.650055 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:09.650055 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.650055 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:09.650055 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.650055 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:09.650952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.650952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:09.650952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:09.650952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:09.650952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.650952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:09.650952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:09.650952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.650952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:09.650952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:09.650952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:09.650952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.651236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:09.651462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.651462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:09.652658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:09.653042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:09.653474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:09.653474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:09.653474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:14.082035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:14.082501 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:14.082564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:14.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:14.128391 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.128809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.129740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:14.129740 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.129740 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:14.129740 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:14.129740 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.129740 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:14.129740 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.129740 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:14.131001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.131001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:14.131001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:14.131001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:14.131001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.131001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:14.131001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:14.131001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.131001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:14.131001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:14.131001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:14.131001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.131374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:14.131618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:14.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.132273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:14.132273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:14.132273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.132273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:14.132273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.132273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:14.590283 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:14.590574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:14.590723 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:14.592070 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.592261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.592531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.592618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.592834 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.593037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.593228 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.593399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.593653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.593653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.594200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:14.594200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.594200 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:14.594200 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:14.594200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.594200 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:14.594200 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.594200 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:14.595089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.595089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:14.595089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:14.595089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:14.595089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.595089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:14.595089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:14.595089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.595089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:14.595089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:14.595089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:14.595089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.595391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:14.595686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.595686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:14.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:14.596734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.597512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:14.597512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.597512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:14.612420 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:14.612702 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:14.612801 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:14.612939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:56:14.613368 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.613573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.614062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:14.614062 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.614062 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:14.614062 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:14.614062 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.614062 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:14.614062 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.614062 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:14.614827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:14.615364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:14.615364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:14.615364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.615364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:14.615364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:14.615364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.615364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:14.615364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.615364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:14.615956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.615956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:14.615956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:14.615956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:14.615956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.615956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:14.615956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:14.615956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:56:14.615956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:14.615956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:14.615956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.615956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:14.616359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:14.616707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.616707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:14.616707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.616707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:14.617450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:14.618021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:14.618543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.618543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:14.618543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:14.618543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.618543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:14.618543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.618543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:14.628467 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:14.628672 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:14.628750 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:14.628855 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:14.629244 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.629483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.629847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:14.629847 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.629847 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:14.629847 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:14.629847 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.629847 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:14.629847 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.629847 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:14.630568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:14.631035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:14.631035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:14.631035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.631035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:14.631035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:14.631035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.631035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:14.631035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.631035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:14.631672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.631672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:14.631672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:14.631672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:14.631672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.631672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:14.631672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:14.631672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:14.631672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:14.631672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.631672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:14.632081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:14.632428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.632428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:14.632428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.632428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.632903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:14.633476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:14.634113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.634113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:14.634113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.634113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:14.646899 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:14.647152 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:14.647263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:14.647397 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:14.647761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.647973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:14.648295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.648676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:14.648676 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.648676 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:14.648676 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:14.648676 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.648676 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:14.648676 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.648676 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:14.649470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.649470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:14.649470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:14.649470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:14.649470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.649470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:14.649470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:14.649470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.649470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:14.649470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:14.649470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:14.649470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.649864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:14.650279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.650279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:14.650661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:14.651171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:14.651728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:14.651728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:14.651728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:19.082694 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:19.084314 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:19.084314 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:19.085281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:19.089805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=240 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 00:56:19.156009 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.156723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.157276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:19.157276 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.157276 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:19.157276 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:19.157276 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.157276 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:19.157276 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.157276 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:19.158275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.073 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.158275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:19.158275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:19.158275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:19.158275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.158275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:19.158275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:19.158275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.158275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:19.158275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:19.158275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:19.158275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.159259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:19.160166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:19.161184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.162706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:19.162706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:19.162706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.162706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:19.162706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.162706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:19.590827 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:19.591095 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:19.591239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:19.592785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.593036 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.593134 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.593282 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.593415 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.593582 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.593739 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.593837 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.593943 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.594091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.594262 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.594400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.594957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:19.594957 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.594957 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:19.594957 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:19.594957 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.594957 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:19.594957 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.594957 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:19.595900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.595900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:19.595900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:19.595900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:19.595900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.595900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:19.595900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:19.595900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.595900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:19.595900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:19.595900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:19.595900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.596565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:19.597369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:19.598296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.599440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:19.599440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:19.599440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.599440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:19.599440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.599440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:19.612822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:19.613093 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:19.613180 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:19.613322 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:56:19.617303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.617487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.617908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:19.617908 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.617908 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:19.617908 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:19.617908 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.617908 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:19.617908 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.617908 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:19.618499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:19.619130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:19.619724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:19.620355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:19.620355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:19.620355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:19.620355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:19.620355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:19.620355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.620355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:19.620355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:19.620355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.620355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:19.620355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.620355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:19.620921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:19.621715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:19.622480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.622480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:19.622480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:19.622480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.622480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:19.622480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.622480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:19.628873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:19.629182 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:19.629287 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:19.629398 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:19.629727 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.629972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.630704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:19.630704 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.630704 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:19.630704 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:19.630704 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.630704 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:19.630704 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.630704 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:19.631570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:19.631927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:19.631927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:19.631927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.631927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:19.631927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:19.631927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.631927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:19.631927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.631927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:19.632384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.632384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:19.632384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:19.632384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:19.632384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.632384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:19.632384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:19.632384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:19.632384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:19.632384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.632384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:19.632620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:19.632854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.632854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:19.632854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.632854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.633840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:19.634146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:19.634483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.634483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:19.634483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.634483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:19.647334 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:19.647707 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:19.647793 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:19.647937 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:19.648298 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.648519 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:19.649123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.649578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:19.649578 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.649578 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:19.649578 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:19.649578 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.649578 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:19.649578 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.649578 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:19.650280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.650280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:19.650280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:19.650280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:19.650280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.650280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:19.650280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:19.650280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.650280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:19.650280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:19.650280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:19.650280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.650570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:19.650861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.650861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:19.651080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:19.651803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:19.652470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:19.652470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:19.652470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:24.084240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:24.085557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:24.085790 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:24.086652 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:24.128563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:24.155263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.155454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.156115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.156115 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.156115 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.156115 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.156115 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.156115 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.156115 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.156115 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.157012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.070 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.157012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.157012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:24.157012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:24.157012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:24.157012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:24.157012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.157012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.157012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.157012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.157012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.157012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.157297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.157521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.157521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:24.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:24.158280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.158638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:24.158638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.158638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:24.158638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:24.158638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:24.166636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 09 00:56:24.169266 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Number of inconsistencies found at create/update: networks=10, security_groups=3, routers=5, router_ports=4 {{(pid=89255) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 09 00:56:24.169266 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=12 {{(pid=89255) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 09 00:56:24.169266 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource b3e28628-ad68-4ca3-b181-b39c995a69e0 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.200567 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, resource={'id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'name': 'public', 'tenant_id': '5a148cfd27844ee6ad0d79cb077e4985', 'project_id': '5a148cfd27844ee6ad0d79cb077e4985', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['17135a44-456c-43be-8f7d-7714aa3731cf', 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-09T00:51:13Z', 'updated_at': '2026-06-09T00:51:20Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.201112 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.201112 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-6a91ff68-89ba-411b-b54f-ff58910954a3, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.201299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.201804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.201804 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.201804 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.201804 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.201804 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.201804 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.201804 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.201804 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.203874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource b3e28628-ad68-4ca3-b181-b39c995a69e0 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.203874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.203874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.203874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.203874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.203874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.203874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 00:56:24.203874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.203874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.203874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.203874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.204179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.204441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.204441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.204441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.204441 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 05419c49-9db5-4d5d-8bbe-5a7016c12d7e (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.245569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-05419c49-9db5-4d5d-8bbe-5a7016c12d7e, resource={'id': '05419c49-9db5-4d5d-8bbe-5a7016c12d7e', 'name': 'shared', 'tenant_id': '5a148cfd27844ee6ad0d79cb077e4985', 'project_id': '5a148cfd27844ee6ad0d79cb077e4985', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['612c5f15-747e-4ea8-a2d1-d0bbb8f1bc11'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:52:24Z', 'updated_at': '2026-06-09T00:52:25Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 44634}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.245804 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-05419c49-9db5-4d5d-8bbe-5a7016c12d7e, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.246059 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-1c288345-932b-445f-8f63-edb47405877c, external_ids_update=None, columns={'tag': 44634}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.246254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.246790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.246790 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.246790 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.246790 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.246790 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.246790 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.246790 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.246790 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.248309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 05419c49-9db5-4d5d-8bbe-5a7016c12d7e (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.248309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.248309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.248309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.248309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.248309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.248309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 00:56:24.248309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.248309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.248309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.248309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.248561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.248852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.248852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.248852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.248852 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 6a9060d6-7861-4a93-8cbd-ee75725785c8 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.286124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-6a9060d6-7861-4a93-8cbd-ee75725785c8, resource={'id': '6a9060d6-7861-4a93-8cbd-ee75725785c8', 'name': 'tempest-ServerActionsTestOtherA-903393236-network', 'tenant_id': '99072531364b4b5e9afe8cff14d9d414', 'project_id': '99072531364b4b5e9afe8cff14d9d414', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['8c04ff16-b7d9-45ab-96de-c6a369b66065'], 'standard_attr_id': 56, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:53:45Z', 'updated_at': '2026-06-09T00:53:47Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 18599}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.286371 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-6a9060d6-7861-4a93-8cbd-ee75725785c8, col_values=(('external_ids', {'neutron:network_name': 'tempest-ServerActionsTestOtherA-903393236-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.286595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-d7a5f405-42cb-41fb-88a2-ff76706d0dbf, external_ids_update=None, columns={'tag': 18599}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.286685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.287262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.287262 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.287262 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.287262 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.287262 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.287262 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.287262 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.287262 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.288151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 6a9060d6-7861-4a93-8cbd-ee75725785c8 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.288151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.288151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.288151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.288151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.288151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.288151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 00:56:24.288151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.288151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.288151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.288151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.288414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.288715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.288715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.288715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.288715 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource ea80c75f-fd3b-4b54-bb09-85070426a97b (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.317125 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-ea80c75f-fd3b-4b54-bb09-85070426a97b, resource={'id': 'ea80c75f-fd3b-4b54-bb09-85070426a97b', 'name': 'tempest-TaggedBootDevicesTest-1116291205-network', 'tenant_id': '846903b6585348e4a3b22f2ccdbaa27e', 'project_id': '846903b6585348e4a3b22f2ccdbaa27e', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['4db7c5a1-7bff-40af-b994-603fd10b8ac0'], 'standard_attr_id': 89, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:53:46Z', 'updated_at': '2026-06-09T00:53:48Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 42794}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.317375 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-ea80c75f-fd3b-4b54-bb09-85070426a97b, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedBootDevicesTest-1116291205-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.317517 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-51ed83e3-738f-4182-80a8-f22856e25f9b, external_ids_update=None, columns={'tag': 42794}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.317606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.318170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.318170 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.318170 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.318170 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.318170 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.318170 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.318170 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.318170 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.318916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource ea80c75f-fd3b-4b54-bb09-85070426a97b (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.318916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.318916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.318916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.318916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.318916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.318916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 00:56:24.318916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.318916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.318916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.318916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.319413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.319858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.319858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.319858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.319858 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource dc15bda6-1b51-4a9e-af84-ac570d199c11 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.356935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-dc15bda6-1b51-4a9e-af84-ac570d199c11, resource={'id': 'dc15bda6-1b51-4a9e-af84-ac570d199c11', 'name': 'private', 'tenant_id': '5a148cfd27844ee6ad0d79cb077e4985', 'project_id': '5a148cfd27844ee6ad0d79cb077e4985', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['65e82083-c074-4bc8-b13e-2eeabecc299d', '6f142de2-8acf-46b2-af54-c34cf691eff5'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:51:08Z', 'updated_at': '2026-06-09T00:51:11Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 62028}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.357236 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-dc15bda6-1b51-4a9e-af84-ac570d199c11, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.357460 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-77925902-9e1b-4f7c-868e-70a5d73ee785, external_ids_update=None, columns={'tag': 62028}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.357543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.358058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.358058 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.358058 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.358058 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.358058 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.358058 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.358058 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.358058 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.358925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource dc15bda6-1b51-4a9e-af84-ac570d199c11 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.358925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.358925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.358925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.358925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.358925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.358925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 00:56:24.358925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.358925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.358925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.358925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.359341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.359794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.359794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.359794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.359794 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 216826ab-ea33-4f2f-9fc3-e7211b5ac50a (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.389035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-216826ab-ea33-4f2f-9fc3-e7211b5ac50a, resource={'id': '216826ab-ea33-4f2f-9fc3-e7211b5ac50a', 'name': 'tempest-TaggedBootDevicesTest_v242-1428789944-network', 'tenant_id': '9e098d86a61244178806d213d6e921ef', 'project_id': '9e098d86a61244178806d213d6e921ef', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['838edde2-3c10-45c6-8747-6525c778ba53'], 'standard_attr_id': 318, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:55:22Z', 'updated_at': '2026-06-09T00:55:23Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 29278}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.389035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-216826ab-ea33-4f2f-9fc3-e7211b5ac50a, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedBootDevicesTest_v242-1428789944-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.389259 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-2a93ea5d-46ec-4c6b-afc7-28ea8f4f0b22, external_ids_update=None, columns={'tag': 29278}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.389259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.391572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.391572 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.391572 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.391572 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.391572 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.391572 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.391572 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.391572 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.392740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 216826ab-ea33-4f2f-9fc3-e7211b5ac50a (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.392740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.392740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.392740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.392740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.392740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.392740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 00:56:24.392740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.392740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.392740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.392740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.393086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.393454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.393454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.393454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.393454 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource c73f108e-fba1-458b-8a91-bf250c4e6487 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.430177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c73f108e-fba1-458b-8a91-bf250c4e6487, resource={'id': 'c73f108e-fba1-458b-8a91-bf250c4e6487', 'name': 'tempest-TaggedBootDevicesTest-112416997-network', 'tenant_id': 'e6a19215d2c84db0a318c5267af91c11', 'project_id': 'e6a19215d2c84db0a318c5267af91c11', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['0f04b843-f0bc-4bf1-a150-c8e8f20ca006'], 'standard_attr_id': 159, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:54:04Z', 'updated_at': '2026-06-09T00:54:05Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 23357}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.430498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c73f108e-fba1-458b-8a91-bf250c4e6487, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedBootDevicesTest-112416997-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.430855 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-171bfc4a-9ee2-4889-b6f0-6dbf99f39e3d, external_ids_update=None, columns={'tag': 23357}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.431006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.431721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.431721 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.431721 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.431721 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.431721 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.431721 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.431721 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.431721 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.433021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource c73f108e-fba1-458b-8a91-bf250c4e6487 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.433021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.433021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.433021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.433021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.433021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.433021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 00:56:24.433021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.433021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.433021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.433021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.434243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.434243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.434243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.434243 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource d6e9dc1a-65a9-4c4a-9e68-44e04f6c64d7 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.468887 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-d6e9dc1a-65a9-4c4a-9e68-44e04f6c64d7, resource={'id': 'd6e9dc1a-65a9-4c4a-9e68-44e04f6c64d7', 'name': 'tempest-AttachVolumeShelveTestJSON-534449568-network', 'tenant_id': '667e1f7213ac4bfb887ac7c181e497a6', 'project_id': '667e1f7213ac4bfb887ac7c181e497a6', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['84461394-0fd9-4902-a5a2-dd6c3a187d4b'], 'standard_attr_id': 292, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:54:43Z', 'updated_at': '2026-06-09T00:54:44Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 45310}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.469120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-d6e9dc1a-65a9-4c4a-9e68-44e04f6c64d7, col_values=(('external_ids', {'neutron:network_name': 'tempest-AttachVolumeShelveTestJSON-534449568-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.469442 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-66b9d70a-949d-4c32-9336-a3c7db5aa66a, external_ids_update=None, columns={'tag': 45310}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.469576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.472046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.472046 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.472046 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.472046 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.472046 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.472046 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.472046 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.472046 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.473363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource d6e9dc1a-65a9-4c4a-9e68-44e04f6c64d7 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.473363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.473363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.473363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.473363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.473363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.473363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 00:56:24.473363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.473363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.473363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.473363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.473616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.474039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.474039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.474039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.474039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource bb2e777d-2c10-49c6-acb7-5403759974e0 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.513574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-bb2e777d-2c10-49c6-acb7-5403759974e0, resource={'id': 'bb2e777d-2c10-49c6-acb7-5403759974e0', 'name': 'tempest-device-tagging-net1-2074530562', 'tenant_id': '846903b6585348e4a3b22f2ccdbaa27e', 'project_id': '846903b6585348e4a3b22f2ccdbaa27e', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['530f3099-8dfd-4f36-b5c9-3fff0bc2b3b1'], 'standard_attr_id': 258, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:54:28Z', 'updated_at': '2026-06-09T00:54:32Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 14852}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.513574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-bb2e777d-2c10-49c6-acb7-5403759974e0, col_values=(('external_ids', {'neutron:network_name': 'tempest-device-tagging-net1-2074530562', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-707fc686-7a62-4d8c-b4e9-864cdf9ca063, external_ids_update=None, columns={'tag': 14852}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource bb2e777d-2c10-49c6-acb7-5403759974e0 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.513820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.514350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.518124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.518124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.518124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.518124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.518124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.518124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.518124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.518124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.518124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.518124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.518124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.518124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.519366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 52b3b81c-361c-443d-bee8-db09e7d47ad3 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.539945 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-52b3b81c-361c-443d-bee8-db09e7d47ad3, resource={'id': '52b3b81c-361c-443d-bee8-db09e7d47ad3', 'name': 'tempest-device-tagging-net2-1775279235', 'tenant_id': '846903b6585348e4a3b22f2ccdbaa27e', 'project_id': '846903b6585348e4a3b22f2ccdbaa27e', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['cf18d836-938a-4d50-a931-1d67165e4f30'], 'standard_attr_id': 263, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:54:31Z', 'updated_at': '2026-06-09T00:54:34Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 38870}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.540536 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-52b3b81c-361c-443d-bee8-db09e7d47ad3, col_values=(('external_ids', {'neutron:network_name': 'tempest-device-tagging-net2-1775279235', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.540954 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-d94dc7b6-7d39-4de8-97bb-8ba1916c8225, external_ids_update=None, columns={'tag': 38870}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.541099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.545511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.545511 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.545511 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.545511 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.545511 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.545511 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.545511 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.545511 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.546911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 52b3b81c-361c-443d-bee8-db09e7d47ad3 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.546911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.546911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.546911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.546911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.546911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.546911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 00:56:24.546911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.546911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.546911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.546911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.547159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.547408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.547408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.547408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.547408 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 0c51ab2a-6c1a-4a20-9842-804cb468fff6 (type: security_groups) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.592887 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:24.609278 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Successfully bumped revision number for resource 0c51ab2a-6c1a-4a20-9842-804cb468fff6 (type: security_groups) to 3 Jun 09 00:56:24.610835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 5d3e3069-65f4-49d9-af62-fe0aac534310 (type: security_groups) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.615237 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:24.630741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:24.646478 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Successfully bumped revision number for resource 5d3e3069-65f4-49d9-af62-fe0aac534310 (type: security_groups) to 3 Jun 09 00:56:24.646644 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource aabaf89c-59f8-4b5a-9359-f6e43a3e7bc0 (type: security_groups) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.650238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:24.671915 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Successfully bumped revision number for resource aabaf89c-59f8-4b5a-9359-f6e43a3e7bc0 (type: security_groups) to 3 Jun 09 00:56:24.672066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 5f6bcb66-c295-420f-887c-25d9a9ca69dd (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.748344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, resource={'id': '5f6bcb66-c295-420f-887c-25d9a9ca69dd', 'name': 'tempest-TaggedBootDevicesTest_v242-1428789944-router', 'project_id': '9e098d86a61244178806d213d6e921ef', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.36'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::23b'}], 'enable_snat': True}, 'gw_port_id': '1b65fd72-cd97-4e40-bec7-ac483b6c1a87', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.36', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::23b', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T00:55:23Z', 'updated_at': '2026-06-09T00:55:28Z', 'revision_number': 3, 'tenant_id': '9e098d86a61244178806d213d6e921ef'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.748633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedBootDevicesTest_v242-1428789944-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.748842 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "1b65fd72-cd97-4e40-bec7-ac483b6c1a87"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.748973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "1b65fd72-cd97-4e40-bec7-ac483b6c1a87"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.749095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.749686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.749686 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.749686 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.749686 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.749686 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.749686 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.749686 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.749686 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.749957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 5f6bcb66-c295-420f-887c-25d9a9ca69dd. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.751122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 5f6bcb66-c295-420f-887c-25d9a9ca69dd (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.751122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.751122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.751122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.751122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.751122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.751122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 00:56:24.751122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 00:56:24.751122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 00:56:24.751122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 00:56:24.751122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.751542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.751970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.751970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.751970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.751970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.751970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.751970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.751970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.751970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.751970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.751970 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource d605d268-7264-4349-9633-1e26c9359915 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.826720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-d605d268-7264-4349-9633-1e26c9359915, resource={'id': 'd605d268-7264-4349-9633-1e26c9359915', 'name': 'tempest-AttachVolumeShelveTestJSON-534449568-router', 'project_id': '667e1f7213ac4bfb887ac7c181e497a6', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.22'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::3f4'}], 'enable_snat': True}, 'gw_port_id': '7403119d-d0d8-40a6-a5c5-3ab13dd87991', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.22', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::3f4', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T00:54:45Z', 'updated_at': '2026-06-09T00:54:47Z', 'revision_number': 3, 'tenant_id': '667e1f7213ac4bfb887ac7c181e497a6'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.827039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-d605d268-7264-4349-9633-1e26c9359915, columns={'external_ids': {'neutron:router_name': 'tempest-AttachVolumeShelveTestJSON-534449568-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.827234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "7403119d-d0d8-40a6-a5c5-3ab13dd87991"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.827394 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "7403119d-d0d8-40a6-a5c5-3ab13dd87991"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.827520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.828132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.828132 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.828132 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.828132 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.828132 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.828132 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.828132 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.828132 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.828356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router d605d268-7264-4349-9633-1e26c9359915. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.829894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource d605d268-7264-4349-9633-1e26c9359915 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.829894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.829894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.829894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.829894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.829894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.829894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 00:56:24.829894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 00:56:24.829894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 00:56:24.829894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 00:56:24.829894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.830373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.831368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.831368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.831368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.831368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.831368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.831368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.831368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.831368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.831368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.831368 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource c6d696f4-5c75-4f06-8e5d-91b703c21757 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:24.918217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, resource={'id': 'c6d696f4-5c75-4f06-8e5d-91b703c21757', 'name': 'tempest-TaggedBootDevicesTest-112416997-router', 'project_id': 'e6a19215d2c84db0a318c5267af91c11', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.89'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::2bb'}], 'enable_snat': True}, 'gw_port_id': '3ed3dc93-41bd-4490-8fcf-5f83a6aaae5c', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.89', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::2bb', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T00:54:09Z', 'updated_at': '2026-06-09T00:54:15Z', 'revision_number': 3, 'tenant_id': 'e6a19215d2c84db0a318c5267af91c11'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.918449 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedBootDevicesTest-112416997-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.918449 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "3ed3dc93-41bd-4490-8fcf-5f83a6aaae5c"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.918449 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "3ed3dc93-41bd-4490-8fcf-5f83a6aaae5c"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:24.918449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.918449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:24.918449 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.918449 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:24.918449 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:24.918739 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.918739 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:24.918739 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.918739 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:24.918739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router c6d696f4-5c75-4f06-8e5d-91b703c21757. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.918739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource c6d696f4-5c75-4f06-8e5d-91b703c21757 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.918739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:24.918739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:24.918739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:24.918739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:24.918739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:24.919094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:24.919552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:24.919894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:24.919894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:24.919894 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 8dc4b509-18fd-4b08-87a0-8be15626b4e0 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:25.001451 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, resource={'id': '8dc4b509-18fd-4b08-87a0-8be15626b4e0', 'name': 'tempest-TaggedBootDevicesTest-1116291205-router', 'project_id': '846903b6585348e4a3b22f2ccdbaa27e', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.163'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::34b'}], 'enable_snat': True}, 'gw_port_id': '4d3c82a2-3872-4d38-a74d-c832c5c0a327', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.163', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::34b', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T00:53:53Z', 'updated_at': '2026-06-09T00:54:01Z', 'revision_number': 3, 'tenant_id': '846903b6585348e4a3b22f2ccdbaa27e'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.001697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedBootDevicesTest-1116291205-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.001906 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "4d3c82a2-3872-4d38-a74d-c832c5c0a327"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.002074 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "4d3c82a2-3872-4d38-a74d-c832c5c0a327"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.002214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.002799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:25.002799 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.002799 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:25.002799 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:25.002799 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.002799 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:25.002799 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.002799 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:25.002981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 8dc4b509-18fd-4b08-87a0-8be15626b4e0. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.004237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 8dc4b509-18fd-4b08-87a0-8be15626b4e0 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.004237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:25.004237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:25.004237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:25.004237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:25.004237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:25.004237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 00:56:25.004237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 00:56:25.004237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 00:56:25.004237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 00:56:25.004237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:25.004508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:25.004765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.004765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:25.004765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.004765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:25.004765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:25.004765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.004765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:25.004765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.004765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:25.004765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:25.087978 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, resource={'id': 'd4aaf6ce-5920-48d0-b4ab-b1ca4a5933be', 'name': 'tempest-ServerActionsTestOtherA-903393236-router', 'project_id': '99072531364b4b5e9afe8cff14d9d414', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.90'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::2d'}], 'enable_snat': True}, 'gw_port_id': '70399ac3-e09c-4dd6-94fb-3abe59f401e0', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.90', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::2d', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T00:53:50Z', 'updated_at': '2026-06-09T00:53:57Z', 'revision_number': 3, 'tenant_id': '99072531364b4b5e9afe8cff14d9d414'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.088431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, columns={'external_ids': {'neutron:router_name': 'tempest-ServerActionsTestOtherA-903393236-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.088431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "70399ac3-e09c-4dd6-94fb-3abe59f401e0"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.088559 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "70399ac3-e09c-4dd6-94fb-3abe59f401e0"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.088680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.089265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:25.089265 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.089265 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:25.089265 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:25.089265 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.089265 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:25.089265 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.089265 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:25.089605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.090532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.090532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:25.090532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:25.090532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:25.090532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:25.090532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:25.090532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 00:56:25.090532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 00:56:25.090532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 00:56:25.090532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 00:56:25.090532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:25.090973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:25.091533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.091533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:25.091533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.091533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:25.091533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:25.091533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.091533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:25.091533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.091533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:25.091533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 3ed3dc93-41bd-4490-8fcf-5f83a6aaae5c (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:25.207856 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-3ed3dc93-41bd-4490-8fcf-5f83a6aaae5c, resource={'id': '3ed3dc93-41bd-4490-8fcf-5f83a6aaae5c', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:f8:73:36', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'c6d696f4-5c75-4f06-8e5d-91b703c21757', 'device_owner': 'network:router_gateway', 'standard_attr_id': 184, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.89'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::2bb'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T00:54:09Z', 'updated_at': '2026-06-09T00:54:32Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.208167 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-3ed3dc93-41bd-4490-8fcf-5f83a6aaae5c, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::2bb/64', '172.24.5.89/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.208616 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=3ed3dc93-41bd-4490-8fcf-5f83a6aaae5c, lrouter_port=lrp-3ed3dc93-41bd-4490-8fcf-5f83a6aaae5c, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.208616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.209200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:25.209200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.209200 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:25.209200 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:25.209200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.209200 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:25.209200 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.209200 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:25.210337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 3ed3dc93-41bd-4490-8fcf-5f83a6aaae5c (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.210337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:25.210337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:25.210337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:25.210337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:25.210337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:25.210337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 00:56:25.210337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:25.210337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.210337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.210337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:25.210658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.211388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:25.211388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.211388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:25.211388 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 4d3c82a2-3872-4d38-a74d-c832c5c0a327 (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:25.314440 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-4d3c82a2-3872-4d38-a74d-c832c5c0a327, resource={'id': '4d3c82a2-3872-4d38-a74d-c832c5c0a327', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:99:02:2d', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '8dc4b509-18fd-4b08-87a0-8be15626b4e0', 'device_owner': 'network:router_gateway', 'standard_attr_id': 116, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.163'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::34b'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T00:53:53Z', 'updated_at': '2026-06-09T00:53:57Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.314683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-4d3c82a2-3872-4d38-a74d-c832c5c0a327, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::34b/64', '172.24.5.163/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.314904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=4d3c82a2-3872-4d38-a74d-c832c5c0a327, lrouter_port=lrp-4d3c82a2-3872-4d38-a74d-c832c5c0a327, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.315124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.315712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:25.315712 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.315712 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:25.315712 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:25.315712 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.315712 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:25.315712 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.315712 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:25.318319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 4d3c82a2-3872-4d38-a74d-c832c5c0a327 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.318319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:25.318319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:25.318319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:25.318319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:25.318319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:25.318319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 00:56:25.318319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:25.318319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.318319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.318319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:25.318819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.319223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:25.319223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.319223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:25.319223 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 7403119d-d0d8-40a6-a5c5-3ab13dd87991 (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:25.410072 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-7403119d-d0d8-40a6-a5c5-3ab13dd87991, resource={'id': '7403119d-d0d8-40a6-a5c5-3ab13dd87991', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:9f:3d:8b', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'd605d268-7264-4349-9633-1e26c9359915', 'device_owner': 'network:router_gateway', 'standard_attr_id': 298, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.22'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::3f4'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T00:54:45Z', 'updated_at': '2026-06-09T00:54:50Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.410386 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-7403119d-d0d8-40a6-a5c5-3ab13dd87991, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-d605d268-7264-4349-9633-1e26c9359915'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::3f4/64', '172.24.5.22/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.410684 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=7403119d-d0d8-40a6-a5c5-3ab13dd87991, lrouter_port=lrp-7403119d-d0d8-40a6-a5c5-3ab13dd87991, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.410910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.411601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:25.411601 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.411601 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:25.411601 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:25.411601 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.411601 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:25.411601 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.411601 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:25.412401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 7403119d-d0d8-40a6-a5c5-3ab13dd87991 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.412401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:25.412401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:25.412401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:25.412401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:25.412401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:25.412401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 00:56:25.412401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:25.412401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.412401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.412401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:25.412626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.412868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:25.412868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.412868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:25.412868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 70399ac3-e09c-4dd6-94fb-3abe59f401e0 (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 00:56:25.513630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-70399ac3-e09c-4dd6-94fb-3abe59f401e0, resource={'id': '70399ac3-e09c-4dd6-94fb-3abe59f401e0', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:99:ba:69', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'd4aaf6ce-5920-48d0-b4ab-b1ca4a5933be', 'device_owner': 'network:router_gateway', 'standard_attr_id': 103, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.90'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::2d'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T00:53:50Z', 'updated_at': '2026-06-09T00:53:55Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.513862 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-70399ac3-e09c-4dd6-94fb-3abe59f401e0, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::2d/64', '172.24.5.90/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.514159 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=70399ac3-e09c-4dd6-94fb-3abe59f401e0, lrouter_port=lrp-70399ac3-e09c-4dd6-94fb-3abe59f401e0, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.514390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.514896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:25.514896 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.514896 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:25.514896 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:25.514896 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.514896 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:25.514896 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.514896 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:25.515890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 70399ac3-e09c-4dd6-94fb-3abe59f401e0 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.515890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:25.515890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 00:56:25.515890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 00:56:25.515890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 00:56:25.515890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 00:56:25.515890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 00:56:25.515890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:25.515890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.515890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.515890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:25.516135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.516375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:25.516375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.516375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:25.516375 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 16fc5001-6008-4801-bbce-1ffc1a78b7a1 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 00:56:25.516375 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(16fc5001-6008-4801-bbce-1ffc1a78b7a1) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 00:56:25.522820 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 4f6a47b5-d063-44eb-90e9-d982a327f2a0 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 00:56:25.523106 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(4f6a47b5-d063-44eb-90e9-d982a327f2a0) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 00:56:25.528728 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 663883b9-3d1c-4c16-b66d-6830e48a517b (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 00:56:25.528950 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(663883b9-3d1c-4c16-b66d-6830e48a517b) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 00:56:25.535217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 751af773-c5d8-4a04-b2b9-f17004e1a129 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 00:56:25.535217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(751af773-c5d8-4a04-b2b9-f17004e1a129) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 00:56:25.547760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 85720d76-5e69-4909-b0a5-ec77d42a329a (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 00:56:25.548073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(85720d76-5e69-4909-b0a5-ec77d42a329a) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 00:56:25.553457 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 9734857f-0598-43df-a2df-669d3a2eeac3 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 00:56:25.553711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(9734857f-0598-43df-a2df-669d3a2eeac3) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 00:56:25.560283 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 9d7d6958-1628-4ac0-b4e6-8996b826c806 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 00:56:25.560524 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(9d7d6958-1628-4ac0-b4e6-8996b826c806) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 00:56:25.564874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource ad03c1da-183b-46a0-8839-432220acfb86 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 00:56:25.565133 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(ad03c1da-183b-46a0-8839-432220acfb86) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 00:56:25.569463 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource d8f46a08-73f7-4138-8923-401f0bfb6882 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 00:56:25.569712 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(d8f46a08-73f7-4138-8923-401f0bfb6882) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 00:56:25.574261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource eb9fa3f7-505b-4c50-863a-be5ded34d089 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 00:56:25.574479 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(eb9fa3f7-505b-4c50-863a-be5ded34d089) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 00:56:25.578505 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource ef7f9d6a-7c1a-4fd7-a39d-885469b8e24e (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 00:56:25.578728 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(ef7f9d6a-7c1a-4fd7-a39d-885469b8e24e) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 00:56:25.584549 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource f87011f8-86d7-42dc-84d8-13f15f69c1b0 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 00:56:25.584791 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(f87011f8-86d7-42dc-84d8-13f15f69c1b0) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 00:56:25.588446 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_for_inconsistencies finished in 1.430 seconds Jun 09 00:56:25.588657 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:25.588817 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:25.590123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.590274 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.590382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.590485 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.590579 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.590677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.590768 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.590879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.591002 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.591164 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.591311 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.591413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.591899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:25.591899 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.591899 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:25.591899 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:25.591899 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.591899 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:25.591899 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.591899 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:25.592522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.592522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:25.592522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:25.592522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:25.592522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.592522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:25.592522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:25.592522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.592522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.592522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.592522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.592522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.592738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:25.592941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.592941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:25.593288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:25.593553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.593904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:25.593904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.593904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:25.593904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:25.593904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:25.593904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:56:25.593904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.594074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.594311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:25.594311 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.594311 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:25.594311 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:25.594311 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.594311 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:25.594311 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.594311 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:25.594756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:25.595094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.595094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:25.595094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.595094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:25.595094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:25.595094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.595094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:25.595094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.595094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:25.595715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.595715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:25.595715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:25.595715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:25.595715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.595715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:25.595715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:25.595715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:56:25.595715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:25.595715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:25.595715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.595715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:25.596089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:25.596390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.596658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:25.597032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:25.597517 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:25.597517 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.597517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.597775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:25.597775 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.597775 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:25.597775 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:25.597775 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.597775 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:25.597775 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.597775 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:25.598294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:25.598566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.598566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:25.598566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.598566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:25.598566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:25.598566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.598566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:25.598566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.598566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:25.598904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.598904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:25.598904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:25.598904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:25.598904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.598904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:25.598904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:25.598904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:25.598904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:25.598904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.598904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:25.599112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:25.599336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.599336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:25.599336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.599336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.599863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:25.600419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:25.600981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.600981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:25.600981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.600981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:25.600981 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:25.600981 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:25.600981 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:25.600981 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.601290 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:25.601290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.601357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:25.601357 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.601357 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:25.601357 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:25.601357 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.601357 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:25.601357 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.601357 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:25.601989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.601989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:25.601989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:25.601989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:25.601989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.601989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:25.601989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:25.601989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.601989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.601989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.601989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.601989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.602410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:25.602770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.602770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:25.603205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:25.603669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:25.604177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:25.604177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:25.604177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:29.086287 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:29.086287 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:29.086287 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:29.086720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:29.147258 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:29.147454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:29.148013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:29.148013 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:29.148013 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:29.148013 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:29.148013 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:29.148013 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:29.148013 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:29.148013 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:29.149122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.062 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:29.149122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:29.149122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:29.149122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:29.149122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:29.149122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:29.149122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:29.149122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:29.149122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:29.149122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:29.149122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:29.149122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:29.149645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:29.150117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:29.150117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:29.150216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:29.150762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:29.151548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:29.151548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:29.151548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:30.588854 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:30.591449 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:30.593320 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:30.595247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:30.595247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.595247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.595247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.595247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.595426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.595426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.595426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.595426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.595426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.595576 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.601175 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.603843 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:30.603843 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:30.606396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.616289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:30.616289 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.616289 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:30.616289 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:30.616289 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.616289 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:30.616289 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.616289 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:30.616466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.616466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:30.616466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:30.616466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:30.616466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.616466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:30.616466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:30.616466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.616466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:30.616466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:30.616466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:30.616466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.616696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:30.616932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.616932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:30.630231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:30.630935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.631548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:30.631548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.631548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:30.631548 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:30.631548 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:30.631548 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:56:30.633028 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.633273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.635061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:30.635061 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.635061 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:30.635061 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:30.635061 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.635061 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:30.635061 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.635061 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:30.636468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:30.636945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:30.636945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:30.636945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.636945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:30.636945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:30.636945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.636945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:30.636945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.636945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:30.637200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.637200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:30.637200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:30.637200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:30.637200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.637200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:30.637200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:30.637200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:56:30.637200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:30.637200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:30.637200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.637200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:30.637556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:30.637909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:30.638350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:30.638916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:30.639290 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:30.639476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.639662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.641134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:30.641134 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.641134 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:30.641134 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:30.641134 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.641134 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:30.641134 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.641134 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:30.643281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:30.644175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:30.646296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:30.646848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:30.646848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:30.646848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:30.646848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:30.646848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.646848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:30.646848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:30.646848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.646848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:30.646848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.646848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:30.646848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:30.647927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.648524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:30.649404 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:30.649404 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:30.649404 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:30.649404 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.649404 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:30.649404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:30.654292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:30.654976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.655502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:30.655971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:30.656802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:30.656802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:30.656802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:30.656802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:30.656802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:30.656802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:30.656802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:30.656802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:30.656802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:30.656802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:34.086161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:34.086631 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:34.086631 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:34.087155 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:34.140560 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:34.144561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:34.145236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:34.145236 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:34.145236 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:34.145236 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:34.145236 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:34.145236 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:34.145236 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:34.145236 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:34.146232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.059 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:34.146232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:34.146232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:34.146232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:34.146232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:34.146232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:34.146232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:34.146232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:34.146232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:34.146232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:34.146232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:34.146232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:34.146515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:34.146821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:34.146821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:34.147327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:34.147694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:34.148087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:34.148087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:34.148087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:35.591488 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:35.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:35.592002 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:35.593316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.593316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.593483 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.593613 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.593747 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.593898 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.594076 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.594217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.594370 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.594502 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.594636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.594776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.595329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:35.595329 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.595329 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:35.595329 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:35.595329 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.595329 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:35.595329 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.595329 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:35.596054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.596054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:35.596054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:35.596054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:35.596054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.596054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:35.596054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:35.596054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.596054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:35.596054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:35.596054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:35.596054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.596492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:35.596920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.596920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:35.597130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:35.597533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.597889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:35.597889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.597889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:35.629063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:35.629357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:35.629466 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:35.629598 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:56:35.629950 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.630183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.630701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:35.630701 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.630701 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:35.630701 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:35.630701 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.630701 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:35.630701 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.630701 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:35.631526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:35.631973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:35.631973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:35.631973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.631973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:35.631973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:35.631973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.631973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:35.631973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.631973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:35.632471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.632471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:35.632471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:35.632471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:35.632471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.632471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:35.632471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:35.632471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:56:35.632471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:35.632471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:35.632471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.632471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:35.632754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:35.633082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.633082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:35.633082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.633082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:35.633602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:35.634120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:35.634837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.634837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:35.634837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:35.634837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.634837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:35.634837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.634837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:35.638053 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:35.638589 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:35.638691 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:35.638804 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:35.639425 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.639625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.640044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:35.640044 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.640044 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:35.640044 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:35.640044 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.640044 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:35.640044 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.640044 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:35.640596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:35.641185 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:35.641185 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:35.641185 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.641185 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:35.641185 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:35.641185 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.641185 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:35.641185 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.641185 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:35.641581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.641581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:35.641581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:35.641581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:35.641581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.641581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:35.641581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:35.641581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:35.641581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:35.641581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.641581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:35.641794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:35.642039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.642039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:35.642039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.642039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.642465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:35.642793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:35.643127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.643127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:35.643127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.643127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:35.645008 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:35.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:35.645414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:35.645609 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:35.645829 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.646035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:35.646373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.646751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:35.646751 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.646751 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:35.646751 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:35.646751 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.646751 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:35.646751 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.646751 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:35.647504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.647504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:35.647504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:35.647504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:35.647504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.647504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:35.647504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:35.647504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.647504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:35.647504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:35.647504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:35.647504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.647822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:35.648088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.648088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:35.648239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:35.648544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:35.648873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:35.648873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:35.648873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:39.091528 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:39.093470 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:39.093470 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:39.095237 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:39.167285 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:39.167285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:39.169579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:39.169579 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:39.169579 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:39.169579 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:39.169579 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:39.169579 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:39.169579 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:39.169579 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:39.171591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.076 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:39.171591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:39.171591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:39.171591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:39.171591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:39.171591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:39.171591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:39.171591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:39.171591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:39.171591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:39.171591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:39.171591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:39.171895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:39.172150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:39.172434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:39.172735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:39.172735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:39.172735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:39.172735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:39.172735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:39.172735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:40.591923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:40.592474 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:40.592474 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:40.593600 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.593834 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.594005 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.594132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.594275 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.594413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.594520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.594677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.594810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.594936 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.595124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.595263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.600505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:40.600505 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.600505 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:40.600505 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:40.600505 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.600505 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:40.600505 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.600505 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:40.601249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.601249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:40.601249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:40.601249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:40.601249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.601249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:40.601249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:40.601249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.601249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:40.601249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:40.601249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:40.601249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.601609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:40.601966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:40.602414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.603015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:40.603015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:40.603015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.603015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:40.603015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.603015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:40.629548 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:40.629999 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:40.630591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:40.630591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:56:40.632931 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.632931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.634506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:40.634506 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.634506 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:40.634506 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:40.634506 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.634506 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:40.634506 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.634506 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:40.635295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:40.635710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:40.635710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:40.635710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.635710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:40.635710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:40.635710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.635710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:40.635710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.635710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:40.636005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.636005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:40.636005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:40.636005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:40.636005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.636005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:40.636005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:40.636005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:56:40.636005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:40.636005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:40.636005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.636005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:40.636383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:40.637165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.637165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:40.637165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.637165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:40.639875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:40.640453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:40.640974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.640974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:40.640974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:40.640974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.640974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:40.640974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.640974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:40.641245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:40.641776 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:40.641848 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:40.642000 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:40.642500 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.642765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.645294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.645935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:40.647873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.647873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:40.647873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:40.647873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:40.647873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.647873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:40.647873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:40.647873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:40.647873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:40.647873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.647873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:40.648336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:40.650343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:40.650890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:40.651457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:40.651457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:40.651457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:40.651457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:40.651457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.651457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:40.651457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:40.651457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.651457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:40.651457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.651457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:40.651457 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:40.653054 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:40.653659 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:40.653659 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:40.654799 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.656765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:40.657058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:40.659963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:40.660621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.661061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:40.661540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:40.662139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:40.662139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:40.662139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:40.662139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:40.662139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:40.662139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:40.662139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:40.662139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:40.662139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:40.662139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:44.092922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:44.095424 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:44.095627 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:44.096212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:44.153604 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:44.153881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:44.154708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:44.154708 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:44.154708 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:44.154708 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:44.154708 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:44.154708 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:44.154708 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:44.154708 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:44.155491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.059 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:44.155491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:44.155491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:44.155491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:44.155491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:44.155491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:44.155491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:44.155491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:44.155491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:44.155491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:44.155491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:44.155491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:44.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:44.156748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:44.157745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:44.158282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:44.158282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:44.158282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:44.158282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:44.158282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:44.158282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:45.593446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:45.593446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:45.593446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:45.595351 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.595578 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.595744 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.595843 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.595956 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.596067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.596170 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.596382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.596420 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8dc4b509-18fd-4b08-87a0-8be15626b4e0, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.596684 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.596798 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.596909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.597403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:45.597403 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.597403 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:45.597403 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:45.597403 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.597403 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:45.597403 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.597403 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:45.598253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.598253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:45.598253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:45.598253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:45.598253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.598253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:45.598253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:45.598253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.598253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:45.598253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:45.598253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:45.598253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.598547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:45.598761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.598761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:45.598800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:45.599074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.599416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:45.599416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.599416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:45.632446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:45.632446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:45.632446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:45.632601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:56:45.632890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.633005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.633647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:45.633647 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.633647 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:45.633647 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:45.633647 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.633647 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:45.633647 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.633647 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:45.634870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:45.637368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:45.640755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:45.641369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:45.641369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:45.641369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:45.641369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:45.641369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:45.641369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.641369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:45.641369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:45.641369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.641369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:45.641369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.641369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:45.641752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:45.643864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:45.644580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.644580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:45.644580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:45.644580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.644580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:45.644580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.644580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:45.644580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:45.644580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:45.644580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:45.645123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:45.645123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.645123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.645123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:45.645123 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.645123 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:45.645123 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:45.645123 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.645123 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:45.645123 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.645123 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:45.645123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:45.645533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:45.646299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:45.647176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:45.648171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:45.648171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:45.648171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.648171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:45.648171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:45.648171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.648171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:45.648171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.648171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:45.648171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.648171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:45.648171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.648452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.649250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:45.653056 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:45.655326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:45.655454 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:45.655638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:45.655918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.656158 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:45.656445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.656900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:45.656900 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.656900 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:45.656900 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:45.656900 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.656900 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:45.656900 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.656900 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:45.657896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.657896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:45.657896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:45.657896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:45.657896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.657896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:45.657896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:45.657896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.657896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:45.657896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:45.657896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:45.657896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.658226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:45.658518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.658518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:45.659725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:45.660130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:45.660531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:45.660531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:45.660531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:49.095792 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:49.096609 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:49.096609 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:49.097336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:49.155259 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:49.156672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:49.157061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:49.157061 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:49.157061 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:49.157061 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:49.157061 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:49.157061 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:49.157061 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:49.157061 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:49.157919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.061 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:49.157919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:49.157919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:49.157919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:49.157919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:49.157919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:49.157919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:49.157919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:49.157919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:49.157919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:49.157919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:49.157919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:49.158602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:49.158925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:49.158925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:49.160763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:49.162118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:49.162578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:49.162578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:49.162578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:50.594612 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:50.594612 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:50.594612 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:50.599220 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.601051 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-c6d696f4-5c75-4f06-8e5d-91b703c21757, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.601702 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.603182 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.603182 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.603182 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.603182 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.609384 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.609384 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.609384 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.609384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.612700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:50.612700 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.612700 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:50.612700 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:50.612700 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.612700 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:50.612700 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.612700 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:50.615313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.018 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.615313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:50.615313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:50.615313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:50.615313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.615313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:50.615313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:50.615313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.615313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:50.615313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:50.615313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:50.615313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.615705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:50.616057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.616057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:50.616987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:50.617686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.618308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:50.618308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.618308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:50.632614 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:50.634206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:50.634206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:50.634206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:56:50.634616 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.634889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.635571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:50.635571 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.635571 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:50.635571 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:50.635571 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.635571 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:50.635571 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.635571 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:50.636498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:50.636794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:50.636794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:50.636794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.636794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:50.636794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:50.636794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.636794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:50.636794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.636794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:50.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:50.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:50.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:50.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:50.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:50.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:56:50.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:50.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:50.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:50.638583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:50.638808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.638808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:50.638808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.638808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:50.639895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:50.640224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:50.640492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.640492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:50.640492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:50.640492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.640492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:50.640492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.640492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:50.642063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:50.642428 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:50.642554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:50.642704 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:50.643745 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.644001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.648218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:50.648843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:50.649351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.649741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:50.650125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:50.650571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:50.651102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:50.651102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.651102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:50.651102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:50.651102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.651102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:50.651102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.651102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:50.656409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:50.656830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:50.657009 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:50.657534 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:50.659658 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.660614 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:50.660614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.660912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:50.660912 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.660912 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:50.660912 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:50.660912 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.660912 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:50.660912 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.660912 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:50.663225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.663225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:50.663225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:50.663225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:50.663225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.663225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:50.663225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:50.663225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.663225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:50.663225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:50.663225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:50.663225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.663668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:50.664075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.664075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:50.665295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:50.665878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:50.666518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:50.666518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:50.666518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:54.096357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:54.096918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:54.096918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:54.097586 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:54.182861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:54.183136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:54.185632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:54.185632 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:54.185632 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:54.185632 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:54.185632 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:54.185632 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:54.185632 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:54.185632 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:54.186398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.089 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:54.186398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:54.186398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:54.186398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:54.186398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:54.186398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:54.186398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:54.186398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:54.186398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:54.186398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:54.186398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:54.186398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:54.186782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:54.187053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:54.187053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:54.187359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:54.187715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:54.188249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:54.188249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:54.188249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:55.594448 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:55.594914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:55.594914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:55.596301 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.596441 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.596551 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.597727 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.598818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.598818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.598818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.598818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.598818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.598818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.599016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:55.599016 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.599016 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:55.599016 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:55.599016 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.599016 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:55.599016 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.599016 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:55.599838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.599838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:55.599838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:55.599838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:55.599838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.599838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:55.599838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:55.599838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.599838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:55.599838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:55.599838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:55.599838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.600137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:55.600433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.600433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:55.600844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:55.601205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.601582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:55.601582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.601582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:55.633433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:55.634683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:55.634683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:55.634683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:56:55.634683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.634869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:55.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:55.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:55.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:55.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:55.637442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:55.637751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:55.637751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:55.637751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.637751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:55.637751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:55.637751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.637751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:55.637751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.637751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:55.638020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.638020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:55.638020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:55.638020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:55.638020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.638020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:55.638020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:55.638020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:56:55.638020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:55.638020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:55.638020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.638020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:55.638433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:55.638685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.638685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:55.638685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.638685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:55.638818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:55.639133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:55.639519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.639519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:55.639519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:55.639519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.639519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:55.639519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.639519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:55.642565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:55.642894 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:55.642997 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:55.643115 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:55.644271 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.644271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.648118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:56:55.648692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:55.648966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.649252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:55.649523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:55.650182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:55.650572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:55.650572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.650572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:55.650572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:55.650572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.650572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:55.650572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.650572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:55.656893 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:55.657325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:55.657458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:55.657636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:55.658693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.659421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:55.660255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.662474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:55.662474 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.662474 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:55.662474 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:55.662474 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.662474 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:55.662474 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.662474 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:55.663204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.663204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:55.663204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:55.663204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:55.663204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.663204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:55.663204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:55.663204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.663204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:55.663204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:55.663204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:55.663204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.663467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:55.663679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.663679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:55.663873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:55.664178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:55.664538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:55.664538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:55.664538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:56:59.096821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:56:59.098765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:56:59.098765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:56:59.098765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:56:59.137940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:56:59.138169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:59.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:56:59.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:59.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:56:59.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:56:59.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:59.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:56:59.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:59.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:56:59.139868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:59.139868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:56:59.139868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:59.139868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:56:59.139868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:59.139868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:59.139868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:59.139868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:59.139868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:59.139868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:59.139868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:56:59.139868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:59.140449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:56:59.141070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:59.141070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:56:59.142253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:56:59.142862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:56:59.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:56:59.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:56:59.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:00.595470 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:00.597089 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:00.597089 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:00.597392 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.597633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.597800 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.597961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.598159 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.598376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.598537 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.598606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.598808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.598899 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.599080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:00.601642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:00.602577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.604019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:00.604391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:00.604873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:00.604873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:00.604873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:00.604873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.604873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:00.604873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:00.604873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.604873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:00.604873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.604873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:00.634688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:00.635353 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:00.635353 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:00.635353 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:57:00.636427 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.636427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.639229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:00.639738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:00.640000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.640264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:00.640502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.640502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:00.641239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:00.641662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:00.641983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.641983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:00.641983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:00.641983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.641983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:00.641983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.641983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:00.643268 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:00.643430 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:00.643544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:00.643683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:00.644161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.644406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.645183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:00.645183 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.645183 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:00.645183 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:00.645183 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.645183 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:00.645183 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.645183 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:00.647612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:00.647916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:00.647916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:00.647916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.647916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:00.647916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:00.647916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.647916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:00.647916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.647916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:00.649065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.649065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:00.649065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:00.649065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:00.649065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.649065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:00.649065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:00.649065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:00.649065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:00.649065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.649065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:00.649352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:00.649617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.649617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:00.649617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.649617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.652801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:00.653170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:00.653535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.653535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:00.653535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.653535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:00.657317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:00.657857 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:00.657905 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:00.658081 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:00.658531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.658774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:00.659749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.659749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:00.659749 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.659749 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:00.659749 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:00.659749 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.659749 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:00.659749 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.659749 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:00.660515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.660515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:00.660515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:00.660515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:00.660515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.660515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:00.660515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:00.660515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.660515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:00.660515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:00.660515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:00.660515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.660796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:00.661037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.661037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:00.662445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:00.663262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:00.663653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:00.663653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:00.663653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:04.097354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:04.098090 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:04.098090 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:04.101217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:04.443909 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:04.444155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:04.444815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:04.444815 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:04.444815 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:04.444815 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:04.444815 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:04.444815 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:04.444815 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:04.444815 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:04.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.347 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:04.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:04.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:04.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:04.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:04.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:04.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:04.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:04.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:04.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:04.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:04.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:04.447932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:04.448226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:04.448597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:04.448962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:04.448962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:04.448962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:04.448962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:04.448962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:04.448962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:05.596380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:05.596946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:05.596946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:05.597567 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.597724 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.597851 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.597972 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.598102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.598275 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.598457 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.598514 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.598634 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.598754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.598905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.599425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:05.599425 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.599425 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:05.599425 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:05.599425 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.599425 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:05.599425 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.599425 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:05.600005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.600005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:05.600005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:05.600005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:05.600005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.600005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:05.600005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:05.600005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.600005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:05.600005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:05.600005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:05.600005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.600336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:05.600623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.600623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:05.600707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:05.601083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.601594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:05.601594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.601594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:05.635409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:05.635819 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:05.635940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:05.636073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:57:05.636530 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.636725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.637341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:05.637341 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.637341 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:05.637341 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:05.637341 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.637341 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:05.637341 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.637341 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:05.637930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:05.638429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:05.638429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:05.638429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.638429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:05.638429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:05.638429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.638429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:05.638429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.638429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:05.638697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.638697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:05.638697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:05.638697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:05.638697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.638697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:05.638697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:05.638697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:57:05.638697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:05.638697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:05.638697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.638697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:05.639070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:05.639450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:05.639707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:05.640041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:05.640041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:05.640041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:05.640041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:05.640041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:05.640041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.640041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:05.640041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:05.640041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.640041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:05.640041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.640041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:05.643572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:05.643921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:05.644032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:05.644169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:05.644529 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.644743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.645240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:05.645240 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.645240 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:05.645240 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:05.645240 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.645240 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:05.645240 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.645240 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:05.645817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:05.646172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:05.646172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:05.646172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.646172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:05.646172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:05.646172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.646172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:05.646172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.646172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:05.646469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.646469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:05.646469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:05.646469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:05.646469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.646469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:05.646469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:05.646469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:05.646469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:05.646469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.646469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:05.647014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:05.647606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:05.648020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:05.648601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:05.648601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:05.648601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:05.648601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:05.648601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.648601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:05.648601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:05.648601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.648601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:05.648601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.648601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:05.657872 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:05.658215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:05.658342 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:05.658529 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:05.658874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.659166 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:05.659588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.660205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:05.660205 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.660205 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:05.660205 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:05.660205 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.660205 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:05.660205 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.660205 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:05.660984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.660984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:05.660984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:05.660984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:05.660984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.660984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:05.660984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:05.660984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.660984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:05.660984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:05.660984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:05.660984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.661485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:05.661826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:05.662476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:05.663040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:05.663040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:05.663040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:05.663040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:05.663040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:05.663040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:09.098132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:09.100101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:09.100101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:09.100101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:09.153880 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:09.155046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:09.155568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:09.155568 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:09.155568 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:09.155568 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:09.155568 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:09.155568 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:09.155568 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:09.155568 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:09.156514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:09.156514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:09.156514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:09.156514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:09.156514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:09.156514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:09.156514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:09.156514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:09.156514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:09.156514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:09.156514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:09.156514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:09.156763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:09.156999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:09.156999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:09.157591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:09.157872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:09.158365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:09.158365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:09.158365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:10.596863 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:10.597540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:10.597540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:10.598949 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.599911 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.600054 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.600320 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.601153 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.601394 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.601874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.601874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.601874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.602253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.602464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.604625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:10.604625 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.604625 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:10.604625 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:10.604625 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.604625 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:10.604625 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.604625 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:10.605928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.605928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:10.605928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:10.605928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:10.605928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.605928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:10.605928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:10.605928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.605928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:10.605928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:10.605928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:10.605928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.606427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:10.606842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:10.607317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.607813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:10.607813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:10.607813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.607813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:10.607813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.607813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:10.635935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:10.636896 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:10.636896 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:10.636896 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:57:10.637829 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.638418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.641539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:10.641539 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.641539 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:10.641539 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:10.641539 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.641539 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:10.641539 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.641539 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:10.642561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:10.643130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:10.643130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:10.643130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.643130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:10.643130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:10.643130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.643130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:10.643130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.643130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:10.644214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.644214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:10.644214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:10.644214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:10.644214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.644214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:10.644214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:10.644214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:57:10.644214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:10.644214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:10.644214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.644214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:10.644671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:10.645046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.645046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:10.645046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.645046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:10.651122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:10.652061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:10.652664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.652664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:10.652664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:10.652664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.652664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:10.652664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.652664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:10.652664 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:10.653937 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:10.654107 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:10.654301 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:10.655451 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.656498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.657718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:10.657718 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.657718 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:10.657718 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:10.657718 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.657718 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:10.657718 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.657718 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:10.659436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:10.659915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:10.659915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:10.659915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.659915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:10.659915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:10.659915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.659915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:10.659915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.659915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:10.659915 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:10.661351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.661351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:10.661351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:10.661351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:10.661351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.661351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:10.661351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:10.661351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:10.661351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:10.661351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.661351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:10.661798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:10.662352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:10.662789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:10.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:10.664272 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:10.664272 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.665780 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:10.667554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.670120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:10.670120 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.670120 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:10.670120 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:10.670120 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.670120 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:10.670120 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.670120 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:10.671414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.671414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:10.671414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:10.671414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:10.671414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.671414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:10.671414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:10.671414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.671414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:10.671414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:10.671414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:10.671414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.671819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:10.672265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.672265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:10.672587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:10.673033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:10.673602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:10.673602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:10.673602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:14.098987 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:14.103591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:14.103904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:14.104832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:14.190660 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:14.190877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.088 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:14.192637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:14.193108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:14.193383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:14.193383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:14.193383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:14.193383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:14.193383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:14.193383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:14.193757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:14.194373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:14.194734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:14.194734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:14.194734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:15.598779 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:15.598779 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:15.598779 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:15.599401 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.599636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.599784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.599919 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.600058 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.600982 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.601039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.601288 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.601425 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.601623 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.601958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.602203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.606335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:15.606335 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.606335 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:15.606335 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:15.606335 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.606335 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:15.606335 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.606335 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:15.607907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.607907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:15.607907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:15.607907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:15.607907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.607907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:15.607907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:15.607907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.607907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:15.607907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:15.607907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:15.607907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.608253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:15.608776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.608776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:15.611523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:15.612772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.615815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:15.615815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.615815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:15.637597 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:15.638149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:15.638758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:15.638985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:57:15.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.640915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.641623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:15.641623 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.641623 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:15.641623 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:15.641623 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.641623 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:15.641623 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.641623 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:15.644408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:15.645264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:15.645264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:15.645264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.645264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:15.645264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:15.645264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.645264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:15.645264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.645264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:15.647536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.647536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:15.647536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:15.647536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:15.647536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.647536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:15.647536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:15.647536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:57:15.647536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:15.647536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:15.647536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.647536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:15.648521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:15.649672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.649672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:15.649672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.649672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:15.653423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:15.655024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:15.655382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.655382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:15.655382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:15.655382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.655382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:15.655382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.655382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:15.656506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:15.656948 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:15.657250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:15.657546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:15.658535 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.659629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.660004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:15.660004 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.660004 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:15.660004 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:15.660004 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.660004 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:15.660004 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.660004 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:15.662753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:15.663267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:15.663267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:15.663267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.663267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:15.663267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:15.663267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.663267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:15.663267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.663267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:15.666075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.666075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:15.666075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:15.666075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:15.666075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.666075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:15.666075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:15.666075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:15.666075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:15.666075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.666075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:15.666343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:15.666642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.666642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:15.666642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.666642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:15.667731 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.671097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:15.671657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:15.672075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.672075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:15.672075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.672075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:15.672922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:15.673544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:15.674156 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:15.675730 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.675992 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:15.676361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:15.683224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:15.684429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.685000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:15.685000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:15.685000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.685000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:15.685000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.685000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:15.686946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:15.687446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:15.687953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:15.687953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:15.687953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:19.105971 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:19.110672 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:19.110672 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:19.111367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:19.113418 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=942,neutron:UPDATE=3,neutron:DELETE=12 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 00:57:19.198802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:19.198998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:19.199653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:19.199653 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:19.199653 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:19.199653 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:19.199653 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:19.199653 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:19.199653 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:19.199653 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:19.201842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.090 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:19.201842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:19.201842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:19.201842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:19.201842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:19.201842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:19.201842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:19.201842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:19.201842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:19.201842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:19.201842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:19.201842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:19.202280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:19.202654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:19.203136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:19.203693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:19.203693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:19.203693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:19.203693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:19.203693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:19.203693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:20.597861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:20.598791 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:20.598791 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:20.601018 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.601018 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.601018 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.601018 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.601018 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.601379 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.602264 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.602749 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.602749 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.604742 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.604742 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.604742 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.604742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:20.608809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:20.609610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.610011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:20.610507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:20.611249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:20.611249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:20.611249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:20.611249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.611249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:20.611249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:20.611249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.611249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:20.611249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.611249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:20.639831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:20.639831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:20.639831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:20.639831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:57:20.640403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.640461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.642859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:20.642859 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.642859 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:20.642859 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:20.642859 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.642859 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:20.642859 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.642859 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:20.645787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:20.646349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:20.646349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:20.646349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.646349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:20.646349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:20.646349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.646349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:20.646349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.646349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:20.648891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.648891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:20.648891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:20.648891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:20.648891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.648891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:20.648891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:20.648891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:57:20.648891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:20.648891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:20.648891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.648891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:20.650103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:20.651089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:20.651748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:20.653869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:20.653869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:20.653869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:20.653869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:20.653869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:20.653869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.653869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:20.653869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:20.653869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.653869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:20.653869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.653869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:20.656978 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:20.661080 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:20.661080 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:20.661080 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:20.661080 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.661080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.661400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:20.661400 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.661400 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:20.661400 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:20.661400 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.661400 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:20.661400 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.661400 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:20.663490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:20.664008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:20.664008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:20.664008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.664008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:20.664008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:20.664008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.664008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:20.664008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.664008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:20.667372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.667372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:20.667372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:20.667372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:20.667372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.667372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:20.667372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:20.667372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:20.667372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:20.667372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.667372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:20.667920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:20.668458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.668458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:20.668458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.668458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.670497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:20.671107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:20.671806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.671806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:20.671806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.671806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:20.672667 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:20.673117 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:20.673341 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:20.674377 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:20.676489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.676729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:20.677141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.677935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:20.677935 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.677935 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:20.677935 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:20.677935 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.677935 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:20.677935 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.677935 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:20.678957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.678957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:20.678957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:20.678957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:20.678957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.678957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:20.678957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:20.678957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.678957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:20.678957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:20.678957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:20.678957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.679351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:20.679950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.679950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:20.680007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:20.680449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:20.680881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:20.680881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:20.680881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:24.106430 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:24.106960 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:24.106960 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:24.107584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:24.165301 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:24.165803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:24.167331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:24.167331 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:24.167331 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:24.167331 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:24.167331 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:24.167331 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:24.167331 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:24.167331 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:24.169362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.060 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:24.169362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:24.169362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:24.169362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:24.169362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:24.169362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:24.169362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:24.169362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:24.169362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:24.169362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:24.169362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:24.169362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:24.169660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:24.169937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:24.169937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:24.170778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:24.171226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:24.171657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:24.171657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:24.171657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:25.598644 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:25.600638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:25.600638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:25.600985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.601111 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.601234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.601359 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.601468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.601565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.602743 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.602886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.603003 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.603117 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.603248 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.603341 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.603444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.604491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:25.604491 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.604491 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:25.604491 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:25.604491 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.604491 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:25.604491 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.604491 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:25.605872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.605872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:25.605872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:25.605872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:25.605872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.605872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:25.605872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:25.605872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.605872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:25.605872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:25.605872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:25.605872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.606278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:25.606598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.606598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:25.608026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:25.608577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.609166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:25.609166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.609166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:25.638922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:25.639722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:25.639722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:25.639722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:57:25.640329 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.640533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.644456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:25.645285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:25.645712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.646083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:25.646437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.646437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:25.647914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:25.648451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:25.649043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.649043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:25.649043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:25.649043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.649043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:25.649043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.649043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:25.657476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:25.657731 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:25.657840 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:25.657979 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:25.659521 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.659747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.660386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:25.660386 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.660386 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:25.660386 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:25.660386 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.660386 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:25.660386 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.660386 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:25.661423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:25.661822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:25.661822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:25.661822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.661822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:25.661822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:25.661822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.661822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:25.661822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.661822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:25.663533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.663533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:25.663533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:25.663533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:25.663533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.663533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:25.663533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:25.663533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:25.663533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:25.663533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.663533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:25.663807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:25.664230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:25.664520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:25.664908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:25.664908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:25.664908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:25.664908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:25.664908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.664908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:25.664908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:25.664908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.664908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:25.664908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.664908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:25.673106 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:25.673510 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:25.673586 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:25.673755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:25.674489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.674674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:25.674932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.675370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:25.675370 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.675370 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:25.675370 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:25.675370 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.675370 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:25.675370 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.675370 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:25.675951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.675951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:25.675951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:25.675951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:25.675951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.675951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:25.675951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:25.675951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.675951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:25.675951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:25.675951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:25.675951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.676282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:25.676532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.676532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:25.676739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:25.677076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:25.677445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:25.677445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:25.677445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:29.106899 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:29.107807 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:29.107918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:29.108745 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:29.184752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:29.185329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:29.189260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:29.189260 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:29.189260 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:29.189260 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:29.189260 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:29.189260 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:29.189260 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:29.189260 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:29.189664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.080 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:29.189664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:29.189664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:29.189664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:29.189664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:29.189664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:29.189664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:29.189664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:29.189664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:29.189664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:29.189664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:29.189664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:29.190149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:29.190629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:29.191333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:29.192111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:29.192111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:29.192111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:29.192111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:29.192111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:29.192111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:30.599764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:30.600408 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:30.600408 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:30.602804 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.603009 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.603222 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.603399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.603574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.603733 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.603884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.604067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.604268 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.604383 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.604504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.604669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.604803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:30.607379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:30.607782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.608012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:30.608320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:30.608890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:30.608890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:30.608890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:30.608890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.608890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:30.608890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:30.608890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.608890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:30.608890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.608890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:30.639543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:30.640758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:30.640758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:30.640758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:57:30.641593 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.641798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.645151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:30.646522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:30.646999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.648164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:30.649401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:30.650623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:30.654183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:30.654183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:30.654183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.654183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:30.654183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:30.654183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.654183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:30.654183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.654183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:30.659842 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:30.661116 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:30.661116 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:30.661116 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:30.661878 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.662179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.666213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:30.667296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:30.667725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.668129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:30.668600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.668600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.669794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:30.671147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:30.672470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.672470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:30.672470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.672470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:30.674507 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:30.674507 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:30.676837 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:30.678155 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:30.678971 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.679323 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:30.679724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.680448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:30.680448 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.680448 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:30.680448 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:30.680448 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.680448 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:30.680448 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.680448 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:30.681253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.681253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:30.681253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:30.681253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:30.681253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.681253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:30.681253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:30.681253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.681253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:30.681253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:30.681253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:30.681253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.682167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:30.683056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:30.684417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:30.685467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:30.685467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:30.685467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:30.685467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:30.685467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:30.685467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:34.107918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:34.109589 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:34.109704 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:34.110564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:34.158317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:34.158528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:34.159056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:34.159056 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:34.159056 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:34.159056 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:34.159056 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:34.159056 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:34.159056 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:34.159056 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:34.159834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:34.159834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:34.159834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:34.159834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:34.159834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:34.159834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:34.159834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:34.159834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:34.159834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:34.159834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:34.159834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:34.159834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:34.160214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:34.160614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:34.160614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:34.160714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:34.161244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:34.161819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:34.161819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:34.161819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:35.600279 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:35.600934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:35.601039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:35.602057 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.602257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.602422 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.602586 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.602745 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.602930 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.603092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.603276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.603403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.603504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.603681 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.603810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-2af3d700-fc6e-4523-9933-8fa13f1dceb3, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.603943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.605483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:35.605483 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.605483 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:35.605483 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:35.605483 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.605483 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:35.605483 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.605483 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:35.606108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.606108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:35.606108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:35.606108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:35.606108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.606108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:35.606108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:35.606108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.606108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:35.606108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:35.606108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:35.606108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.606460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:35.606781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:35.607225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.607698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:35.607698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:35.607698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.607698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:35.607698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.607698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:35.640322 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:35.640564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:35.640649 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:35.640748 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:57:35.641060 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.641278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.642051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:35.642051 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.642051 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:35.642051 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:35.642051 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.642051 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:35.642051 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.642051 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:35.642798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:35.643101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:35.643101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:35.643101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.643101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:35.643101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:35.643101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.643101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:35.643101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.643101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:35.643343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.643343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:35.643343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:35.643343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:35.643343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.643343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:35.643343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:35.643343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:57:35.643343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:35.643343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:35.643343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.643343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:35.643566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:35.643787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.643787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:35.643787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.643787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:35.643988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:35.644813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:35.645161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.645161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:35.645161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:35.645161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.645161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:35.645161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.645161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:35.662798 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:35.663221 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:35.663221 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:35.663428 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:35.663795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.664665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.665247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:35.665247 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.665247 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:35.665247 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:35.665247 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.665247 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:35.665247 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.665247 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:35.666115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:35.666994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:35.666994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:35.666994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.666994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:35.666994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:35.666994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.666994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:35.666994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.666994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:35.667406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.667406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:35.667406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:35.667406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:35.667406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.667406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:35.667406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:35.667406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:35.667406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:35.667406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.667406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:35.667699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:35.667964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.667964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:35.667964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.667964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.668697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:35.670508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:35.671284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.671284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:35.671284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.671284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:35.674479 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:35.676393 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:35.676393 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:35.676393 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:35.676393 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.676890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:35.677039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.677676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:35.677676 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.677676 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:35.677676 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:35.677676 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.677676 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:35.677676 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.677676 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:35.680868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.680868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:35.680868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:35.680868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:35.680868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.680868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:35.680868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:35.680868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.680868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:35.680868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:35.680868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:35.680868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.681270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:35.681533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.681533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:35.682031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:35.683253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:35.684286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:35.684286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:35.684286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:39.112297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:39.113590 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:39.113590 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:39.115543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:39.169995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:39.170179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:39.170646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:39.170646 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:39.170646 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:39.170646 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:39.170646 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:39.170646 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:39.170646 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:39.170646 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:39.171526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:39.171526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:39.171526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:39.171526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:39.171526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:39.171526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:39.171526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:39.171526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:39.171526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:39.171526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:39.171526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:39.171526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:39.171960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:39.172474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:39.172474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:39.172561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:39.173071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:39.173662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:39.173662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:39.173662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:40.601113 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:40.601577 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:40.601611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:40.602569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.602731 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.602884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.603050 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.603165 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.603281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.603387 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.603524 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.603632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.603761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.603888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.604232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.604640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:40.604640 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.604640 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:40.604640 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:40.604640 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.604640 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:40.604640 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.604640 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:40.605433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.605433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:40.605433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:40.605433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:40.605433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.605433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:40.605433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:40.605433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.605433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:40.605433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:40.605433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:40.605433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.605706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:40.605938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.605938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:40.606203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:40.606499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.606801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:40.606801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.606801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:40.649687 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:40.650071 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:40.650156 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:40.650248 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:57:40.650606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.650825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.651442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:40.651442 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.651442 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:40.651442 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:40.651442 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.651442 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:40.651442 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.651442 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:40.652422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:40.652700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:40.652700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:40.652700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.652700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:40.652700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:40.652700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.652700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:40.652700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.652700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:40.653355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.653355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:40.653355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:40.653355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:40.653355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.653355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:40.653355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:40.653355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:57:40.653355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:40.653355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:40.653355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.653355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:40.653573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:40.653790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.653790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:40.653790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.653790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:40.654460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:40.654761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:40.655068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.655068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:40.655068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:40.655068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.655068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:40.655068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.655068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:40.663314 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:40.663538 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:40.663677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:40.663833 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:40.664206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.664498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.665099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:40.665099 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.665099 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:40.665099 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:40.665099 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.665099 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:40.665099 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.665099 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:40.665906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:40.666183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:40.666183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:40.666183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.666183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:40.666183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:40.666183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.666183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:40.666183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.666183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:40.666924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.666924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:40.666924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:40.666924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:40.666924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.666924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:40.666924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:40.666924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:40.666924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:40.666924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.666924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:40.667148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:40.667390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.667390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:40.667390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.667390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.668424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:40.668724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:40.669027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.669027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:40.669027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.669027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:40.675559 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:40.675818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:40.675947 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:40.676156 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:40.676465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.676725 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:40.677182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.677833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:40.677833 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.677833 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:40.677833 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:40.677833 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.677833 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:40.677833 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.677833 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:40.678766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.678766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:40.678766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:40.678766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:40.678766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.678766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:40.678766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:40.678766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.678766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:40.678766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:40.678766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:40.678766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.679013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:40.679238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.679238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:40.679890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:40.680233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:40.680542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:40.680542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:40.680542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:44.113408 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:44.117609 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:44.117722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:44.118336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:44.172497 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:44.172748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:44.173330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:44.173330 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:44.173330 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:44.173330 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:44.173330 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:44.173330 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:44.173330 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:44.173330 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:44.173933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:44.173933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:44.173933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:44.173933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:44.173933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:44.173933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:44.173933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:44.173933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:44.173933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:44.173933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:44.173933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:44.173933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:44.174376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:44.174663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:44.174663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:44.174791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:44.175327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:44.175735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:44.175735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:44.175735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:45.603471 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:45.605320 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:45.605320 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:45.607227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.607712 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2e55afe8-0db7-4a2b-ad1f-ecb8915d3555, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.607712 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.607712 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.607830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.607830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.607931 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.608034 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.608126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.608281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-5f6bcb66-c295-420f-887c-25d9a9ca69dd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.608325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.608417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.609418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:45.609418 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.609418 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:45.609418 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:45.609418 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.609418 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:45.609418 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.609418 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:45.609914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.609914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:45.609914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:45.609914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:45.609914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.609914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:45.609914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:45.609914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.609914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:45.609914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:45.609914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:45.609914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.610295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:45.610535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.610535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:45.610836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:45.611210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.611592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:45.611592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.611592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:45.650348 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:45.651136 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:45.651288 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:45.651350 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:57:45.653483 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.657493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.658343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:45.658343 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.658343 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:45.658343 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:45.658343 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.658343 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:45.658343 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.658343 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:45.659048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:45.659480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:45.659480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:45.659480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.659480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:45.659480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:45.659480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.659480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:45.659480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.659480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:45.661387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.661387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:45.661387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:45.661387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:45.661387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.661387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:45.661387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:45.661387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:57:45.661387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:45.661387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:45.661387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.661387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:45.661671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:45.661956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.661956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:45.661956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.661956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:45.664132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:45.664684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:45.665212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.665212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:45.665212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:45.665212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.665212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:45.665212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.665212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:45.665212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:45.665212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:45.665212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:45.665492 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:45.665492 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.665700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.666200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:45.666200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.666200 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:45.666200 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:45.666200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.666200 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:45.666200 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.666200 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:45.666891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:45.667612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:45.668055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:45.668633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:45.668633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:45.668633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:45.668633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:45.668633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.668633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:45.668633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:45.668633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.668633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:45.668633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.668633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:45.668633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:45.668875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.669280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:45.676023 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:45.676777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:45.676777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:45.676868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:45.677606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.677656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:45.678236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.679247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:45.679247 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.679247 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:45.679247 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:45.679247 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.679247 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:45.679247 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.679247 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:45.679872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.679872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:45.679872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:45.679872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:45.679872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.679872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:45.679872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:45.679872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.679872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:45.679872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:45.679872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:45.679872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.681298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:45.681905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:45.682518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:45.683048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:45.683048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:45.683048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:45.683048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:45.683048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:45.683048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:49.117783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:49.118719 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:49.118863 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:49.119517 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:49.171329 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:49.171581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:49.172207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:49.172207 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:49.172207 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:49.172207 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:49.172207 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:49.172207 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:49.172207 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:49.172207 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:49.173104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:49.173104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:49.173104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:49.173104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:49.173104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:49.173104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:49.173104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:49.173104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:49.173104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:49.173104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:49.173104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:49.173104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:49.173438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:49.173721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:49.173721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:49.174541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:49.174932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:49.175335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:49.175335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:49.175335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:50.604782 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:50.605170 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:50.605170 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:50.607360 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.607934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.608229 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.608480 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.608938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.609249 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.609665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.610210 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.610553 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.611581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.612804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:50.612804 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.612804 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:50.612804 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:50.612804 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.612804 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:50.612804 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.612804 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:50.613710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.613710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:50.613710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:50.613710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:50.613710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.613710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:50.613710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:50.613710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.613710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:50.613710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:50.613710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:50.613710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.614443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:50.615010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:50.615370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.616059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:50.616059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:50.616059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.616059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:50.616059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.616059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:50.655511 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:50.656760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:50.656917 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:50.657047 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:57:50.661664 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.665647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.669102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:50.669489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:50.669489 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.669489 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:50.669489 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:50.669489 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.669489 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:50.669489 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.669489 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:50.671553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.015 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:50.672711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:50.673692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:50.674174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:50.674174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:50.674174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:50.674174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:50.674174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:50.674174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.674174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:50.674174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:50.674174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.674174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:50.674174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.674174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:50.675049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:50.675702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:50.677145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.677145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:50.677145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:50.677145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.677145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:50.677145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.677145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:50.677145 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:50.677145 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:50.677145 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:50.677890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.679129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.680951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:50.681205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:50.681205 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.681205 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:50.681205 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:50.681205 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.681205 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:50.681205 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.681205 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:50.681945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:50.682358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:50.682358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:50.682358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.682358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:50.682358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:50.682358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.682358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:50.682358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.682358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:50.682825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.682825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:50.682825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:50.682825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:50.682825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.682825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:50.682825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:50.682825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:50.682825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:50.682825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.682825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:50.683412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:50.683753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.683753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:50.683753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.683753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.684086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:50.684538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:50.685125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.685125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:50.685125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.685125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:50.685125 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:50.685125 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:50.685125 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:50.685340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.685463 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:50.686457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.686911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:50.686911 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.686911 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:50.686911 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:50.686911 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.686911 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:50.686911 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.686911 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:50.687782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.687782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:50.687782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:50.687782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:50.687782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.687782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:50.687782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:50.687782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.687782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:50.687782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:50.687782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:50.687782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.688464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:50.689090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:50.689653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:50.690254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:50.690254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:50.690254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:50.690254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:50.690254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:50.690254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:54.118802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:54.120519 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:54.120519 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:54.120519 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:54.199335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:54.199497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:54.202462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:54.202462 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:54.202462 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:54.202462 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:54.202462 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:54.202462 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:54.202462 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:54.202462 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:54.203354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.083 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:54.203354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:54.203354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:54.203354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:54.203354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:54.203354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:54.203354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:54.203354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:54.203354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:54.203354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:54.203354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:54.203354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:54.203897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:54.204173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:54.204173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:54.204250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:54.204837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:54.205533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:54.205533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:54.205533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:55.605068 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:55.606061 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:55.606061 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:55.607350 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.607350 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.608632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.608632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.608632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.608632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.608632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.609605 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.610147 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-157eba5a-4096-40b2-aaec-6424aa890145, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.610401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.611902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:55.611902 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.611902 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:55.611902 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:55.611902 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.611902 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:55.611902 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.611902 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:55.613285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.613285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:55.613285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:55.613285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:55.613285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.613285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:55.613285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:55.613285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.613285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:55.613285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:55.613285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:55.613285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.613728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:55.614689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.614689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:55.616379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:55.617004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.617525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:55.617525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.617525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:55.661001 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:55.662197 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:55.662291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:55.662497 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:57:55.672059 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.673348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.674003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:55.674003 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.674003 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:55.674003 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:55.674003 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.674003 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:55.674003 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.674003 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:55.674880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:55.675463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:55.675463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:55.675463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.675463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:55.675463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:55.675463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.675463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:55.675463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.675463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:55.678355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.013 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.678355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:55.678355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:55.678355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:55.678355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.678355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:55.678355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:55.678355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:57:55.678355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:55.678355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:55.678355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.678355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:55.678803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:55.679265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:55.679709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:55.680328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:55.680328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:55.680328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:55.680328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:55.680328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:55.680328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.680328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:55.680328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:55.680328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.680328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:55.680328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.680328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:55.680867 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:55.684575 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:55.690873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:55.691092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:55.691255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:55.691707 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.691886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.692511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:55.692511 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.692511 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:55.692511 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:55.692511 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.692511 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:55.692511 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.692511 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:57:55.693543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:57:55.694034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:55.694034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:57:55.694034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.694034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:57:55.694034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:57:55.694034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.694034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:57:55.694034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.694034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:57:55.695581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.695581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:55.695581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:55.695581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:55.695581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.695581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:55.695581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:55.695581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:55.695581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:55.695581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.695581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:55.695960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:55.696404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.696404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:55.696404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.696404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.699411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:55.699933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:55.700546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.700546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:55.700546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.700546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:55.700546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:55.700546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:55.700546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:55.700546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.700816 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:55.701601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.702057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:55.702057 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.702057 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:55.702057 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:55.702057 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.702057 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:55.702057 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.702057 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:55.702976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.702976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:55.702976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:55.702976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:55.702976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.702976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:55.702976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:55.702976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.702976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:55.702976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:55.702976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:55.702976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.703537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:55.703916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.703916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:55.705132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:55.705715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:55.706316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:55.706316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:55.706316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:57:59.120291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:57:59.120291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:57:59.121088 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:57:59.122245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:57:59.180250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:57:59.180531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:59.181326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:57:59.181326 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:59.181326 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:57:59.181326 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:57:59.181326 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:59.181326 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:57:59.181326 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:59.181326 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:57:59.182295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.061 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:59.182295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:57:59.182295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:59.182295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:57:59.182295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:59.182295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:59.182295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:59.182295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:59.182295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:59.182295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:59.182295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:57:59.182295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:59.182750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:57:59.183180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:59.183180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:57:59.183489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:57:59.184066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:57:59.184765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:57:59.184765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:57:59.184765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:00.605923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:00.608506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:00.608629 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:00.609746 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.610005 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-054593a2-24c1-4654-b522-164425e2f0c7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.610155 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.610325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.610473 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.610617 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.610775 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.610916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.611064 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.611216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.611812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:00.611812 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.611812 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:00.611812 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:00.611812 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.611812 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:00.611812 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.611812 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:00.612930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.612930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:00.612930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:00.612930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:00.612930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.612930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:00.612930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:00.612930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.612930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:00.612930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:00.612930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:00.612930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.613567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:00.613911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.613911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:00.614009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:00.614548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.615133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:00.615133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.615133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:00.662339 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:00.662610 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:00.662678 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:00.662905 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:58:00.663310 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.663487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.665968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:00.665968 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.665968 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:00.665968 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:00.665968 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.665968 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:00.665968 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.665968 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:00.667212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:00.667556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:00.667556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:00.667556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.667556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:00.667556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:00.667556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.667556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:00.667556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.667556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:00.667816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.667816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:00.667816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:00.667816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:00.667816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.667816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:00.667816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:00.667816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:58:00.667816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:00.667816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:00.667816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.667816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:00.668080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:00.668314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.668314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:00.668314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.668314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:00.671868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:00.672297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:00.672596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.672596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:00.672596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:00.672596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.672596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:00.672596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.672596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:00.690992 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:00.692307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:00.693347 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:00.693347 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:00.693713 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.693903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.694411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:00.694411 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.694411 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:00.694411 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:00.694411 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.694411 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:00.694411 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.694411 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:00.695183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:00.695511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:00.695511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:00.695511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.695511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:00.695511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:00.695511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.695511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:00.695511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.695511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:00.696110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.696110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:00.696110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:00.696110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:00.696110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.696110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:00.696110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:00.696110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:00.696110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:00.696110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.696110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:00.696370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:00.696610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.696610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:00.696610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.696610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.698247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:00.698821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:00.699266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.699266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:00.699266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.699266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:00.699476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:00.699848 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:00.699980 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:00.700164 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:00.700484 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.701499 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.704495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:00.705183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:00.705543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:00.706270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:00.706270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:00.706270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:00.706270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:00.706270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:00.706270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:00.706270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:00.706270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:00.706270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:00.706270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:00.706270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:00.706270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:04.120213 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:04.122816 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:04.122816 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:04.122816 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:04.166258 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:04.166404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:04.169174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:04.169798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:04.170244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:04.170660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:04.171306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:04.171306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:04.171306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:04.171306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:04.171306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:04.171306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:04.171306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:04.171306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:04.171306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:05.608619 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:05.610465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:05.610465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:05.610553 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.610795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.610987 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.611146 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.611317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.611502 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.611640 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.611874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.612035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.612142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.612647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:05.612647 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.612647 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:05.612647 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:05.612647 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.612647 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:05.612647 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.612647 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:05.614709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.614709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:05.614709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:05.614709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:05.614709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.614709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:05.614709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:05.614709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.614709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:05.614709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:05.614709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:05.614709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.615019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:05.615241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:05.615637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.615973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:05.615973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:05.615973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.615973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:05.615973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.615973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:05.662818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:05.663611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:05.663611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:05.663611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:58:05.663913 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.664292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.667480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:05.668132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:05.668649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.669064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:05.669491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:05.669952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:05.670561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:05.670561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:05.670561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.670561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:05.670561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:05.670561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.670561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:05.670561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.670561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:05.692310 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:05.693301 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:05.693301 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:05.693301 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:05.693591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.693857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.697432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:05.698021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:05.698414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.698749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:05.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:05.699456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:05.699916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:05.699916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.699916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:05.699916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:05.699916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.699916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:05.699916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.699916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:05.699916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:05.700676 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:05.700874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:05.701116 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:05.701611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.701849 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:05.702077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.703428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:05.703428 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.703428 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:05.703428 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:05.703428 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.703428 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:05.703428 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.703428 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:05.704131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.704131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:05.704131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:05.704131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:05.704131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.704131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:05.704131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:05.704131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.704131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:05.704131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:05.704131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:05.704131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.704520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:05.704824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.704824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:05.706023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:05.706481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:05.706977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:05.706977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:05.706977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:09.121809 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:09.122364 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:09.122364 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:09.124830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:09.176502 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:09.176690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:09.177160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:09.177160 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:09.177160 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:09.177160 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:09.177160 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:09.177160 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:09.177160 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:09.177160 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:09.178028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:09.178028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:09.178028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:09.178028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:09.178028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:09.178028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:09.178028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:09.178028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:09.178028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:09.178028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:09.178028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:09.178028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:09.179356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:09.179953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:09.179953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:09.182383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:09.182814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:09.183299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:09.183299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:09.183299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:10.609699 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:10.611318 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:10.611448 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:10.614276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.614510 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.614656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.614797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.614953 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.615123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.615279 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.615431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.615590 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.615754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.615983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.617871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:10.617871 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.617871 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:10.617871 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:10.617871 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.617871 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:10.617871 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.617871 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:10.618760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.618760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:10.618760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:10.618760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:10.618760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.618760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:10.618760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:10.618760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.618760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:10.618760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:10.618760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:10.618760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.619169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:10.619536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.619536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:10.619847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:10.620319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.620808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:10.620808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.620808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:10.663420 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:10.663729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:10.663817 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:10.663921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:58:10.664510 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.664704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.665183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:10.665183 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.665183 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:10.665183 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:10.665183 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.665183 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:10.665183 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.665183 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:10.665753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:10.666119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:10.666119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:10.666119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.666119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:10.666119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:10.666119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.666119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:10.666119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.666119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:10.666635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.666635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:10.666635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:10.666635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:10.666635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.666635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:10.666635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:10.666635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:58:10.666635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:10.666635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:10.666635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.666635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:10.666957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:10.667320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:10.667584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:10.667984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:10.667984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:10.667984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:10.667984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:10.667984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:10.667984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.667984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:10.667984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:10.667984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.667984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:10.667984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.667984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:10.692927 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:10.693295 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:10.693403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:10.693512 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:10.694069 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.694306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.694808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:10.694808 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.694808 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:10.694808 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:10.694808 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.694808 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:10.694808 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.694808 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:10.695572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:10.695838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:10.695838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:10.695838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.695838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:10.695838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:10.695838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.695838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:10.695838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.695838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:10.696153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.696153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:10.696153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:10.696153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:10.696153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.696153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:10.696153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:10.696153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:10.696153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:10.696153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.696153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:10.696379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:10.696976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.696976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:10.696976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.696976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.697107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:10.697495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:10.697853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.697853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:10.697853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.697853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:10.702439 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:10.703873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:10.703873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:10.703873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:10.703873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.704045 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:10.704384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.704979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:10.704979 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.704979 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:10.704979 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:10.704979 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.704979 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:10.704979 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.704979 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:10.705905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.705905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:10.705905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:10.705905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:10.705905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.705905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:10.705905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:10.705905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.705905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:10.705905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:10.705905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:10.705905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.706257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:10.706495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.706495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:10.707113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:10.707453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:10.707789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:10.707789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:10.707789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:14.122383 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:14.122893 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:14.122893 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:14.123639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:14.186694 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:14.186694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:14.187556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:14.187556 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:14.187556 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:14.187556 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:14.187556 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:14.187556 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:14.187556 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:14.187556 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:14.188298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.065 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:14.188298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:14.188298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:14.188298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:14.188298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:14.188298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:14.188298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:14.188298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:14.188298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:14.188298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:14.188298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:14.188298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:14.188770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:14.190159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:14.190712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:14.191214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:14.191214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:14.191214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:14.191214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:14.191214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:14.191214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:15.611462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:15.612031 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:15.612031 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:15.613207 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.613465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.613639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.613789 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.613927 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.614091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.614227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.614373 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.614497 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.614615 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.614743 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.614865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.616783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:15.616783 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.616783 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:15.616783 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:15.616783 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.616783 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:15.616783 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.616783 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:15.617626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.617626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:15.617626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:15.617626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:15.617626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.617626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:15.617626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:15.617626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.617626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:15.617626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:15.617626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:15.617626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.618359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:15.618689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:15.619154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.619776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:15.619776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:15.619776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.619776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:15.619776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.619776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:15.664680 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:15.666532 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:15.666640 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:15.666776 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:58:15.667160 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.667383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.668143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:15.668143 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.668143 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:15.668143 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:15.668143 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.668143 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:15.668143 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.668143 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:15.669604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:15.670160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:15.670160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:15.670160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.670160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:15.670160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:15.670160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.670160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:15.670160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.670160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:15.670563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.670563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:15.670563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:15.670563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:15.670563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.670563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:15.670563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:15.670563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:58:15.670563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:15.670563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:15.670563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.670563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:15.671000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:15.671452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:15.671924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:15.674014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:15.674014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:15.674014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:15.674014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:15.674014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:15.674014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.674014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:15.674014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:15.674014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.674014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:15.674014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.674014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:15.694623 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:15.695028 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:15.695103 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:15.695241 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:15.697050 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.697381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.698222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:15.698222 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.698222 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:15.698222 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:15.698222 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.698222 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:15.698222 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.698222 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:15.699272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:15.699582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:15.699582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:15.699582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.699582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:15.699582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:15.699582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.699582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:15.699582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.699582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:15.701065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.701065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:15.701065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:15.701065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:15.701065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.701065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:15.701065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:15.701065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:15.701065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:15.701065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.701065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:15.701357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:15.701588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.701588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:15.701588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.701588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.703628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:15.704031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:15.704387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.704387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:15.704387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.704387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:15.704387 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:15.705759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:15.705759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:15.705759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:15.706154 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.706395 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:15.706713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.707415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:15.707415 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.707415 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:15.707415 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:15.707415 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.707415 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:15.707415 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.707415 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:15.708917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.708917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:15.708917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:15.708917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:15.708917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.708917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:15.708917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:15.708917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.708917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:15.708917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:15.708917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:15.708917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.709329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:15.710604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:15.711072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:15.712297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:15.712297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:15.712297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:15.712297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:15.712297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:15.712297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:19.123017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:19.123496 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:19.123496 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:19.124143 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:19.149596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=250 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 00:58:19.167356 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:19.167526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:19.168102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:19.168102 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:19.168102 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:19.168102 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:19.168102 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:19.168102 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:19.168102 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:19.168102 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:19.168702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:19.168702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:19.168702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:19.168702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:19.168702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:19.168702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:19.168702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:19.168702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:19.168702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:19.168702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:19.168702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:19.168702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:19.169092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:19.169477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:19.169869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:19.170402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:19.170402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:19.170402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:19.170402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:19.170402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:19.170402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:20.611918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:20.614648 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:20.614648 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:20.617526 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.617784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.618088 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.618292 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.618410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.618526 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.618624 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.618727 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.618899 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.618979 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.619095 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.624215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.624553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:20.624553 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.624553 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:20.624553 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:20.624553 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.624553 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:20.624553 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.624553 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:20.625818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.012 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.625818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:20.625818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:20.625818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:20.625818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.625818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:20.625818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:20.625818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.625818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:20.625818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:20.625818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:20.625818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.626389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:20.626942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.626942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:20.628316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:20.629001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.629677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:20.629677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.629677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:20.667211 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:20.667211 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:20.667211 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:20.667435 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:58:20.667879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.668179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.668760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:20.668760 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.668760 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:20.668760 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:20.668760 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.668760 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:20.668760 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.668760 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:20.669607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:20.670231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:20.670231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:20.670231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.670231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:20.670231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:20.670231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.670231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:20.670231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.670231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:20.671863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.671863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:20.671863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:20.671863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:20.671863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.671863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:20.671863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:20.671863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:58:20.671863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:20.671863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:20.671863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.671863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:20.672662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:20.675840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:20.676655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:20.677574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:20.677574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:20.677574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:20.677574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:20.677574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:20.677574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.677574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:20.677574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:20.677574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.677574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:20.677574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.677574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:20.695409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:20.700722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:20.700722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:20.700722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:20.700722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.700722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.700722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:20.700722 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.700722 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:20.700722 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:20.701024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.701516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.701896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:20.702720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:20.703059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.704623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:20.705133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:20.705133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.705133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:20.705133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.705133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:20.705133 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:20.705336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:20.705336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:20.705491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:20.705822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.706328 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:20.706659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.707206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:20.707206 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.707206 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:20.707206 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:20.707206 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.707206 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:20.707206 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.707206 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:20.708121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.708121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:20.708121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:20.708121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:20.708121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.708121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:20.708121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:20.708121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.708121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:20.708121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:20.708121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:20.708121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.708482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:20.708851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.708851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:20.709413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:20.709903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:20.710384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:20.710384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:20.710384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:24.125934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:24.125934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:24.125934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:24.125934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:24.176395 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:24.177149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:24.178729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:24.178729 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:24.178729 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:24.178729 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:24.178729 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:24.178729 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:24.178729 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:24.178729 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:24.179662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:24.179662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:24.179662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:24.179662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:24.179662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:24.179662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:24.179662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:24.179662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:24.179662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:24.179662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:24.179662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:24.179662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:24.180089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:24.180527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:24.180527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:24.181877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:24.184314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:24.185629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:24.185629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:24.185629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:25.614763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:25.617332 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:25.617332 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:25.622116 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.622336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.622336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.622838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.622838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.622838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.622838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.622838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.623183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.623183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.623183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.623183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.625010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:25.625010 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.625010 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:25.625010 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:25.625010 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.625010 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:25.625010 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.625010 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:25.632019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.632019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:25.632019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:25.632019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:25.632019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.632019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:25.632019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:25.632019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.632019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:25.632019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:25.632019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:25.632019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.632335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:25.632596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.632596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:25.635722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:25.636402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.637345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:25.637345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.637345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:25.667036 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:25.667346 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:25.667635 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:25.667802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:58:25.668139 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.668615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.669301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:25.669301 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.669301 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:25.669301 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:25.669301 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.669301 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:25.669301 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.669301 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:25.670135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:25.670615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:25.670615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:25.670615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.670615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:25.670615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:25.670615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.670615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:25.670615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.670615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:25.670943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.670943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:25.670943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:25.670943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:25.670943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.670943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:25.670943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:25.670943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:58:25.670943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:25.670943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:25.670943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.670943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:25.671370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:25.671697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.671697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:25.671697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.671697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:25.672117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:25.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:25.673099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.673099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:25.673099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:25.673099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.673099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:25.673099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.673099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:25.696034 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:25.697593 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:25.697704 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:25.697801 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:25.701293 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.701539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.702511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:25.702511 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.702511 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:25.702511 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:25.702511 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.702511 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:25.702511 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.702511 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:25.703385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:25.703819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:25.703819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:25.703819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.703819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:25.703819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:25.703819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.703819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:25.703819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.703819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:25.704285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.704285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:25.704285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:25.704285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:25.704285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.704285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:25.704285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:25.704285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:25.704285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:25.704285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.704285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:25.704617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:25.704989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.704989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:25.704989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.704989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.705518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:25.706027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:25.706540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.706540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:25.706540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.706540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:25.706540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:25.706690 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:25.706788 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:25.706904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:25.707295 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.707588 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:25.708008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.709036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:25.709036 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.709036 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:25.709036 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:25.709036 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.709036 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:25.709036 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.709036 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:25.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:25.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:25.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:25.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:25.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:25.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:25.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:25.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:25.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.710662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:25.710996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.710996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:25.711275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:25.711721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:25.712248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:25.712248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:25.712248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:29.126358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:29.126358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:29.126828 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:29.127095 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:29.183545 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:29.183882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:29.184382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:29.184382 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:29.184382 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:29.184382 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:29.184382 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:29.184382 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:29.184382 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:29.184382 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:29.185169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:29.185169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:29.185169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:29.185169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:29.185169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:29.185169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:29.185169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:29.185169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:29.185169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:29.185169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:29.185169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:29.185169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:29.185685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:29.186042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:29.186042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:29.187903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:29.188431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:29.188859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:29.188859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:29.188859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:30.617369 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:30.618358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:30.618492 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:30.619733 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.620013 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.620297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.620490 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.620729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.620904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.624217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.624217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.624217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.624217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.624217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.624217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:30.624544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:30.625041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.625471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:30.625930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:30.626644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:30.626644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:30.626644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:30.626644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.626644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:30.626644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:30.626644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.626644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:30.626644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.626644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:30.667507 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:30.668160 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:30.668294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:30.668410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:58:30.668892 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.669253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.669860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:30.669860 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.669860 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:30.669860 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:30.669860 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.669860 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:30.669860 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.669860 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:30.672205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:30.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:30.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:30.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:30.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:30.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:30.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.672621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:30.675784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.675784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:30.675784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:30.675784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:30.675784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.675784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:30.675784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:30.675784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:58:30.675784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:30.675784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:30.675784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.675784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:30.676463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:30.676692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:30.676982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:30.677396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:30.677396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:30.677396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:30.677396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:30.677396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:30.677396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.677396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:30.677396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:30.677396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.677396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:30.677396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.677396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:30.697767 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:30.698729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:30.698729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:30.698729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:30.699061 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.699275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.700133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:30.700133 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.700133 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:30.700133 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:30.700133 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.700133 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:30.700133 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.700133 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:30.701140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:30.701592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:30.701592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:30.701592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.701592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:30.701592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:30.701592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.701592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:30.701592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.701592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:30.702645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.702645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:30.702645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:30.702645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:30.702645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.702645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:30.702645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:30.702645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:30.702645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:30.702645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.702645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:30.702943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:30.703422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.703422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:30.703422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.703422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.704502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:30.705019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:30.705417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.705417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:30.705417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.705417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:30.707421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:30.707822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:30.708010 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:30.708264 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:30.708589 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.708891 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:30.709287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.710021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:30.710021 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.710021 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:30.710021 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:30.710021 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.710021 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:30.710021 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.710021 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:30.711151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.711151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:30.711151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:30.711151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:30.711151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.711151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:30.711151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:30.711151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.711151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:30.711151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:30.711151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:30.711151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.711564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:30.711936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.711936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:30.712670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:30.713125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:30.713725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:30.713725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:30.713725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:34.125958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:34.126787 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:34.126909 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:34.127708 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:34.189003 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:34.189176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:34.189668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:34.189668 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:34.189668 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:34.189668 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:34.189668 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:34.189668 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:34.189668 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:34.189668 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:34.190318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.063 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:34.190318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:34.190318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:34.190318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:34.190318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:34.190318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:34.190318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:34.190318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:34.190318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:34.190318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:34.190318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:34.190318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:34.190714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:34.191084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:34.191541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:34.192064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:34.192064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:34.192064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:34.192064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:34.192064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:34.192064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:35.620458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:35.621122 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:35.621297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:35.622232 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.622407 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.622604 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.622763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.622916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.623079 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.623265 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.623423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.623571 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.623718 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.623940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.624063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:35.625792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:35.626413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.626692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:35.626943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:35.627372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:35.627372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:35.627372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:35.627372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.627372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:35.627372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:35.627372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.627372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:35.627372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.627372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:35.668609 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:35.669436 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:35.669642 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:35.669802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:58:35.670764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.671789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.673304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:35.673304 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.673304 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:35.673304 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:35.673304 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.673304 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:35.673304 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.673304 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:35.675112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:35.675508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:35.675765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:35.676020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:35.676020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:35.676020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:35.676020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:35.676020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:35.676020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.676020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:35.676020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:35.676020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.676020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:35.676020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.676020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:35.680639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:35.681133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:35.681712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.681712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:35.681712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:35.681712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.681712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:35.681712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.681712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:35.698256 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:35.698665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:35.698784 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:35.698921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:35.699381 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.699671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.700431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:35.700431 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.700431 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:35.700431 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:35.700431 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.700431 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:35.700431 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.700431 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:35.701323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:35.701737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:35.701737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:35.701737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.701737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:35.701737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:35.701737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.701737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:35.701737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.701737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:35.702240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.702240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:35.702240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:35.702240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:35.702240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.702240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:35.702240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:35.702240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:35.702240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:35.702240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.702240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:35.702567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:35.702922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.702922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:35.702922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.702922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.704855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:35.705371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:35.705846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.705846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:35.705846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.705846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:35.707906 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:35.708303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:35.708428 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:35.708582 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:35.709025 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.709316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:35.709635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.710226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:35.710226 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.710226 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:35.710226 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:35.710226 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.710226 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:35.710226 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.710226 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:35.711245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.711245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:35.711245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:35.711245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:35.711245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.711245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:35.711245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:35.711245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.711245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:35.711245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:35.711245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:35.711245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.711579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:35.711901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.711901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:35.712506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:35.712993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:35.713516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:35.713516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:35.713516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:39.126884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:39.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:39.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:39.128005 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:39.170065 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:39.170286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:39.170763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:39.170763 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:39.170763 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:39.170763 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:39.170763 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:39.170763 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:39.170763 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:39.170763 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:39.171601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:39.171601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:39.171601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:39.171601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:39.171601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:39.171601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:39.171601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:39.171601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:39.171601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:39.171601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:39.171601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:39.171601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:39.172240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:39.172616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:39.173055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:39.173556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:39.173556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:39.173556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:39.173556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:39.173556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:39.173556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:40.621400 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:40.622490 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:40.622770 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:40.624632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.624841 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.628358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.628639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.628818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.628973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.629106 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.629257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.629382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.629498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.629615 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.629731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.631513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:40.631513 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.631513 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:40.631513 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:40.631513 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.631513 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:40.631513 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.631513 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:40.634484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.634484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:40.634484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:40.634484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:40.634484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.634484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:40.634484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:40.634484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.634484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:40.634484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:40.634484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:40.634484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.634866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:40.635261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.635261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:40.639257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:40.639792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.640356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:40.640356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.640356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:40.670655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:40.671858 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:40.671935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:40.672050 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:58:40.672512 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.672703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.673800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:40.673800 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.673800 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:40.673800 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:40.673800 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.673800 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:40.673800 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.673800 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:40.674734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:40.675205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:40.675205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:40.675205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.675205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:40.675205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:40.675205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.675205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:40.675205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.675205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:40.675604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.675604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:40.675604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:40.675604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:40.675604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.675604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:40.675604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:40.675604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:58:40.675604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:40.675604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:40.675604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.675604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:40.676066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:40.676481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.676481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:40.676481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.676481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:40.676610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:40.676889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:40.677251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.677251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:40.677251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:40.677251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.677251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:40.677251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.677251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:40.699043 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:40.701549 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:40.701681 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:40.701734 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:40.702469 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.703107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.703510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:40.703510 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.703510 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:40.703510 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:40.703510 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.703510 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:40.703510 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.703510 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:40.705854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:40.706262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:40.706262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:40.706262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.706262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:40.706262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:40.706262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.706262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:40.706262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.706262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:40.706673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.706673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:40.706673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:40.706673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:40.706673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.706673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:40.706673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:40.706673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:40.706673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:40.706673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.706673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:40.706895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:40.707160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.707160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:40.707160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.707160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.707476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:40.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:40.708286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.708286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:40.708286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.708286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:40.708764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:40.710015 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:40.710089 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:40.710363 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:40.710791 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.711231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:40.711423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.712131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:40.712131 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.712131 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:40.712131 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:40.712131 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.712131 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:40.712131 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.712131 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:40.714235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.714235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:40.714235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:40.714235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:40.714235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.714235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:40.714235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:40.714235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.714235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:40.714235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:40.714235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:40.714235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.714599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:40.715020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.715020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:40.716604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:40.717246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:40.717840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:40.717840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:40.717840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:44.127922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:44.130538 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:44.130875 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:44.131735 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:44.216813 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:44.217304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:44.217983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:44.217983 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:44.217983 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:44.217983 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:44.217983 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:44.217983 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:44.217983 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:44.217983 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:44.222838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.087 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:44.222838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:44.222838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:44.222838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:44.222838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:44.222838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:44.222838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:44.222838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:44.222838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:44.222838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:44.222838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:44.222838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:44.223180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:44.223436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:44.223436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:44.224044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:44.224424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:44.224807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:44.224807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:44.224807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:45.622479 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:45.623627 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:45.623799 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:45.627286 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.627803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.627803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.627803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.627975 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.627975 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.628706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.629504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.629655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.629803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.630216 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.630216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.631042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:45.631042 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.631042 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:45.631042 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:45.631042 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.631042 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:45.631042 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.631042 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:45.633824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.633824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:45.633824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:45.633824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:45.633824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.633824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:45.633824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:45.633824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.633824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:45.633824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:45.633824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:45.633824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.634403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:45.634775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.634775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:45.638422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:45.638942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.639591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:45.639591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.639591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:45.671938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:45.673372 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:45.673563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:45.673720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:58:45.681310 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.681517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.682246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:45.682246 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.682246 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:45.682246 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:45.682246 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.682246 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:45.682246 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.682246 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:45.683939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:45.684472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:45.684472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:45.684472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.684472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:45.684472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:45.684472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.684472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:45.684472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.684472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:45.684936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.684936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:45.684936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:45.684936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:45.684936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.684936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:45.684936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:45.684936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:58:45.684936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:45.684936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:45.684936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.684936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:45.685335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:45.687870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:45.688318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:45.689069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:45.689069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:45.689069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:45.689069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:45.689069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:45.689069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.689069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:45.689069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:45.689069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.689069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:45.689069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.689069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:45.702607 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:45.703771 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:45.703919 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:45.704083 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:45.707927 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.708210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.709378 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:45.709735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:45.709735 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.709735 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:45.709735 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:45.709735 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.709735 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:45.709735 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.709735 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:45.711466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:45.711766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:45.711766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:45.711766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.711766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:45.711766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:45.711766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.711766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:45.711766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.711766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:45.713562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.713562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:45.713562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:45.713562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:45.713562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.713562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:45.713562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:45.713562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:45.713562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:45.713562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.713562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:45.716758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:45.717079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:45.717408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:45.717740 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:45.717994 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:45.717994 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.717994 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:45.717994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.719156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:45.719156 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.719156 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:45.719156 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:45.719156 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.719156 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:45.719156 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.719156 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:45.720443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.720443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:45.720443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:45.720443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:45.720443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.720443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:45.720443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:45.720443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.720443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:45.720443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:45.720443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:45.720443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.720843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:45.721265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.721265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:45.721740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:45.722121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:45.722696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:45.722696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:45.722696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:49.132620 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:49.132620 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:49.132620 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:49.132620 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:49.180679 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:49.181619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:49.182056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:49.182056 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:49.182056 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:49.182056 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:49.182056 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:49.182056 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:49.182056 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:49.182056 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:49.183182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:49.183182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:49.183182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:49.183182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:49.183182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:49.183182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:49.183182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:49.183182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:49.183182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:49.183182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:49.183182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:49.183182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:49.183590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:49.183978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:49.183978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:49.184470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:49.184944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:49.185467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:49.185467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:49.185467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:50.625452 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:50.626308 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:50.626408 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:50.627422 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.627665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.627838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.628032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.628154 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.628348 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.628478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.628639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.628791 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.628928 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.629149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.629271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.630966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:50.630966 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.630966 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:50.630966 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:50.630966 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.630966 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:50.630966 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.630966 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:50.631804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.631804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:50.631804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:50.631804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:50.631804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.631804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:50.631804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:50.631804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.631804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:50.631804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:50.631804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:50.631804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.632248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:50.633069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.633069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:50.633575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:50.634370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.634847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:50.634847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.634847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:50.672379 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:50.672795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:50.672925 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:50.673329 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:58:50.673514 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.673879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.674350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:50.674350 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.674350 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:50.674350 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:50.674350 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.674350 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:50.674350 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.674350 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:50.675096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:50.675558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:50.675558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:50.675558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.675558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:50.675558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:50.675558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.675558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:50.675558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.675558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:50.676230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.676230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:50.676230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:50.676230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:50.676230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.676230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:50.676230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:50.676230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:58:50.676230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:50.676230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:50.676230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.676230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:50.677245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:50.677786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:50.678389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:50.678871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:50.678871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:50.678871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:50.678871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:50.678871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:50.678871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.678871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:50.678871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:50.678871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.678871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:50.678871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.678871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:50.703858 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:50.704403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:50.704540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:50.704690 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:50.705089 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.705380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.705937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:50.705937 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.705937 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:50.705937 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:50.705937 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.705937 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:50.705937 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.705937 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:50.708719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:50.709332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:50.709733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:50.709990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:50.709990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:50.709990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:50.709990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:50.709990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.709990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:50.709990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:50.709990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.709990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:50.709990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.709990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:50.709990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:50.710218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:50.716177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:50.716776 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:50.716776 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:50.716992 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:50.718791 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.719165 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:50.719603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.720337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:50.720337 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.720337 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:50.720337 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:50.720337 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.720337 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:50.720337 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.720337 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:50.721325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.721325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:50.721325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:50.721325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:50.721325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.721325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:50.721325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:50.721325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.721325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:50.721325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:50.721325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:50.721325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.721755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:50.722148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.722148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:50.723318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:50.723797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:50.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:50.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:50.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:54.131032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:54.131583 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:54.131638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:54.133203 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:54.207930 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:54.208105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:54.210028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:54.210028 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:54.210028 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:54.210028 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:54.210028 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:54.210028 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:54.210028 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:54.210028 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:54.211808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.078 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:54.211808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:54.211808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:54.211808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:54.211808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:54.211808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:54.211808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:54.211808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:54.211808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:54.211808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:54.211808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:54.211808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:54.212221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:54.212548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:54.212979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:54.215757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:54.215757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:54.215757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:54.215757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:54.215757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:54.215757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:55.626411 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:55.628337 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:55.628337 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:55.629065 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.629372 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.629649 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.629871 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.630123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.630431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.630654 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.630878 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.631135 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.631412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.631668 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.631919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.632529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:55.632529 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.632529 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:55.632529 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:55.632529 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.632529 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:55.632529 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.632529 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:55.634019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.634019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:55.634019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:55.634019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:55.634019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.634019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:55.634019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:55.634019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.634019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:55.634019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:55.634019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:55.634019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.636063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:55.636333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:55.636610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.637035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:55.637035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:55.637035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.637035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:55.637035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.637035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:55.673594 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:55.674495 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:55.674495 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:55.674495 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:58:55.675035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.675365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.675966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:55.675966 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.675966 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:55.675966 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:55.675966 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.675966 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:55.675966 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.675966 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:55.678426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:55.678824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:55.678824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:55.678824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.678824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:55.678824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:55.678824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.678824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:55.678824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.678824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:55.680645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.680645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:55.680645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:55.680645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:55.680645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.680645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:55.680645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:55.680645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:58:55.680645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:55.680645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:55.680645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.680645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:55.681307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:55.681851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:55.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:55.683088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:55.683088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:55.683088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:55.683088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:55.683088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:55.683088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.683088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:55.683088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:55.683088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.683088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:55.683088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.683088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:55.704602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:55.705695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:55.705695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:55.705695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:55.711547 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.711547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.712349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:55.712349 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.712349 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:55.712349 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:55.712349 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.712349 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:55.712349 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.712349 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:58:55.713013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:58:55.713475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:55.713475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:58:55.713475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.713475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:58:55.713475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:58:55.713475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.713475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:58:55.713475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.713475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:58:55.713796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.713796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:55.713796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:55.713796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:55.713796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.713796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:55.713796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:55.713796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:55.713796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:55.713796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.713796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:55.714154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:55.714537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:55.714947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:55.715517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:55.715517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:55.715517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:55.715517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:55.715517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.715517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:55.715517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:55.715517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.715517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:55.715517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.715517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:55.716785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:55.717049 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:55.717115 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:55.717273 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:55.717702 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.717972 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:55.718322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.718919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:55.718919 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.718919 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:55.718919 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:55.718919 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.718919 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:55.718919 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.718919 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:55.719601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.719601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:55.719601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:55.719601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:55.719601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.719601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:55.719601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:55.719601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.719601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:55.719601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:55.719601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:55.719601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.719900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:55.720231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.720231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:55.720376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:55.720703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:55.721053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:55.721053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:55.721053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:58:59.131557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:58:59.132810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:58:59.132939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:58:59.133806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:58:59.187032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:58:59.187433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:59.188447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:58:59.188447 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:59.188447 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:58:59.188447 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:58:59.188447 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:59.188447 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:58:59.188447 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:59.188447 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:58:59.189279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:59.189279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:58:59.189279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:59.189279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:58:59.189279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:59.189279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:59.189279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:59.189279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:59.189279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:59.189279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:59.189279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:58:59.189279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:59.189696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:58:59.190058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:59.190058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:58:59.190313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:58:59.190752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:58:59.191443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:58:59.191443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:58:59.191443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:00.627327 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:00.628098 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:00.628180 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:00.629174 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.629362 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.629504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.629633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.629787 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.629910 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.630067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.630222 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.630354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.630481 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.630607 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.630737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.631265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:00.631265 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.631265 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:00.631265 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:00.631265 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.631265 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:00.631265 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.631265 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:00.631842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.631842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:00.631842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:00.631842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:00.631842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.631842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:00.631842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:00.631842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.631842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:00.631842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:00.631842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:00.631842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.632267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:00.632628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:00.633118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.633673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:00.633673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:00.633673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.633673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:00.633673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.633673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:00.674206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:00.674510 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:00.674572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:00.674694 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:59:00.675043 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.675226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.675682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:00.675682 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.675682 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:00.675682 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:00.675682 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.675682 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:00.675682 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.675682 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:00.676356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:00.676813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:00.676813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:00.676813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.676813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:00.676813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:00.676813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.676813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:00.676813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.676813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:00.677243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.677243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:00.677243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:00.677243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:00.677243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.677243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:00.677243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:00.677243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:59:00.677243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:00.677243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:00.677243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.677243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:00.677612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:00.678086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.678086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:00.678086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.678086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:00.678360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:00.678795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:00.679339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.679339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:00.679339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:00.679339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.679339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:00.679339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.679339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:00.705502 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:00.705782 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:00.705999 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:00.706131 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:00.706445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.706633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.707064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:00.707064 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.707064 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:00.707064 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:00.707064 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.707064 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:00.707064 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.707064 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:00.707664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:00.708226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:00.708226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:00.708226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.708226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:00.708226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:00.708226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.708226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:00.708226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.708226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:00.708526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.708526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:00.708526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:00.708526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:00.708526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.708526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:00.708526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:00.708526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:00.708526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:00.708526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.708526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:00.708890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:00.709266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:00.709731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:00.710054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:00.710054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:00.710054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:00.710054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:00.710054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.710054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:00.710054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:00.710054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.710054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:00.710054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.710054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:00.717775 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:00.718217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:00.718518 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:00.718713 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:00.720622 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.720848 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:00.721213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.721615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:00.721615 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.721615 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:00.721615 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:00.721615 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.721615 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:00.721615 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.721615 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:00.722365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.722365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:00.722365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:00.722365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:00.722365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.722365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:00.722365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:00.722365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.722365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:00.722365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:00.722365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:00.722365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.723597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:00.723966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:00.724428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:00.724938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:00.724938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:00.724938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:00.724938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:00.724938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:00.724938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:04.133507 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:04.135281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:04.135281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:04.139660 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:04.195988 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:04.196118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:04.196670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:04.196670 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:04.196670 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:04.196670 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:04.196670 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:04.196670 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:04.196670 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:04.196670 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:04.197520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.062 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:04.197520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:04.197520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:04.197520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:04.197520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:04.197520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:04.197520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:04.197520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:04.197520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:04.197520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:04.197520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:04.197520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:04.197992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:04.198461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:04.198970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:04.199636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:04.199636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:04.199636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:04.199636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:04.199636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:04.199636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:05.629258 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:05.630685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:05.630685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:05.631769 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.631946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.632280 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.632280 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.632280 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.632493 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.632493 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.632589 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.632691 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.632862 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.632907 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.633039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.634207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:05.634207 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.634207 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:05.634207 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:05.634207 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.634207 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:05.634207 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.634207 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:05.634501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.634501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:05.634501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:05.634501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:05.634501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.634501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:05.634501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:05.634501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.634501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:05.634501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:05.634501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:05.634501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.634970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:05.635338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:05.635786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.636357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:05.636357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:05.636357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.636357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:05.636357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.636357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:05.674679 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:05.675647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:05.675647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:05.675647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:59:05.676141 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.676409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.676927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:05.676927 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.676927 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:05.676927 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:05.676927 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.676927 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:05.676927 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.676927 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:05.677562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:05.677900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:05.677900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:05.677900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.677900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:05.677900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:05.677900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.677900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:05.677900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.677900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:05.678183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.678183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:05.678183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:05.678183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:05.678183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.678183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:05.678183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:05.678183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:59:05.678183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:05.678183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:05.678183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.678183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:05.678440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:05.678771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.678771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:05.678771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.678771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:05.679180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:05.679484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:05.679792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.679792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:05.679792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:05.679792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.679792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:05.679792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.679792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:05.705966 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:05.706250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:05.706376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:05.706520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:05.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.707846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:05.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:05.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:05.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:05.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.710016 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:05.713829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:05.714413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:05.714413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:05.714413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.714413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:05.714413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:05.714413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.714413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:05.714413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.714413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:05.714658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.714658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:05.714658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:05.714658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:05.714658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.714658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:05.714658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:05.714658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:05.714658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:05.714658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.714658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:05.715008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:05.715389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:05.715787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:05.716341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:05.716341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:05.716341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:05.716341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:05.716341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.716341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:05.716341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:05.716341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.716341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:05.716341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.716341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:05.718875 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:05.719287 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:05.719368 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:05.719506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:05.721611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.721611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:05.721611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.721611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:05.721611 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.721611 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:05.721611 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:05.721611 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.721611 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:05.721611 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.721611 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:05.721999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.721999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:05.721999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:05.721999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:05.721999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.721999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:05.721999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:05.721999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.721999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:05.721999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:05.721999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:05.721999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.722395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:05.722749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:05.724037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:05.724663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:05.724663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:05.724663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:05.724663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:05.724663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:05.724663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:09.135124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:09.135565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:09.135565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:09.136081 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:09.179091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:09.179353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:09.183042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:09.183814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:09.184587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:09.185035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:09.185534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:09.185534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:09.185534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:09.185534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:09.185534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:09.185534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:09.185534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:09.185534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:09.185534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:10.630672 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:10.631257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:10.631257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:10.633229 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.633229 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.633229 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.633229 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.633229 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.633508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.633508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.633508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.633508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.633642 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cd0f35b4-46b4-497f-b137-061c5de82f63, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.633681 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.633865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.634422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:10.634422 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.634422 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:10.634422 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:10.634422 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.634422 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:10.634422 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.634422 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:10.635312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.635312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:10.635312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:10.635312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:10.635312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.635312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:10.635312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:10.635312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.635312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:10.635312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:10.635312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:10.635312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.635595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:10.635830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.635830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:10.636479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:10.636874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.637548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:10.637548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.637548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:10.676091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:10.679574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:10.679574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:10.679574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:59:10.679574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.679574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.679574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:10.679574 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.679574 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:10.679574 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:10.680038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.680635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:10.681154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.681645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:10.682060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:10.682594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:10.683684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:10.683684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:10.683684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.683684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:10.683684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:10.683684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.683684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:10.683684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.683684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:10.706384 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:10.706910 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:10.707162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:10.707307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:10.707803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.708081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.709370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:10.709370 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.709370 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:10.709370 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:10.709370 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.709370 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:10.709370 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.709370 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:10.710200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:10.710694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:10.710694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:10.710694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.710694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:10.710694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:10.710694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.710694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:10.710694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.710694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:10.711236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.711236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:10.711236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:10.711236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:10.711236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.711236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:10.711236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:10.711236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:10.711236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:10.711236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.711236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:10.711670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:10.712225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.712225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:10.712225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.712225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.712971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:10.713524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:10.714295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.714295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:10.714295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.714295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:10.720547 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:10.720926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:10.721007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:10.721149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:10.721414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.721657 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:10.721928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.722586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:10.722586 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.722586 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:10.722586 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:10.722586 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.722586 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:10.722586 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.722586 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:10.723260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.723260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:10.723260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:10.723260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:10.723260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.723260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:10.723260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:10.723260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.723260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:10.723260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:10.723260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:10.723260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.723718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:10.724138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:10.724944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:10.725654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:10.725654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:10.725654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:10.725654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:10.725654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:10.725654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:14.137561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:14.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:14.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:14.138697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:14.181789 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:14.181976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:14.184160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:14.184619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:14.184987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:14.185320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:14.185744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:14.185744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:14.185744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:14.185744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:14.185744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:14.185744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:14.185744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:14.185744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:14.185744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:15.631166 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:15.631904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:15.631904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:15.632840 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.632912 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.633035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.633275 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.633275 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-215ca9d0-0e01-47f0-9baf-75d056f0d022, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.633504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.633719 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.633719 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.634004 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.634004 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.634004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.634749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:15.634749 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.634749 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:15.634749 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:15.634749 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.634749 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:15.634749 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.634749 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:15.635554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.635554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:15.635554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:15.635554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:15.635554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.635554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:15.635554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:15.635554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.635554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:15.635554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:15.635554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:15.635554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.636891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:15.637246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:15.637677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.638149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:15.638149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:15.638149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.638149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:15.638149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.638149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:15.678638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:15.678885 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:15.679032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:15.679137 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:59:15.679741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.679962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.680529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:15.680529 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.680529 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:15.680529 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:15.680529 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.680529 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:15.680529 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.680529 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:15.681289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:15.681893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:15.681893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:15.681893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.681893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:15.681893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:15.681893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.681893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:15.681893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.681893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:15.682092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.682092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:15.682092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:15.682092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:15.682092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.682092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:15.682092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:15.682092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:59:15.682092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:15.682092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:15.682092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.682092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:15.682329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:15.682545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.682545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:15.682545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.682545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:15.683032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:15.684071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:15.684615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.684615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:15.684615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:15.684615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.684615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:15.684615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.684615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:15.707039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:15.707485 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:15.707584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:15.707723 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:15.708234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.708457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.708995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:15.708995 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.708995 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:15.708995 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:15.708995 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.708995 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:15.708995 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.708995 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:15.709620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:15.709884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:15.709884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:15.709884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.709884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:15.709884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:15.709884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.709884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:15.709884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.709884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:15.710281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.710281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:15.710281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:15.710281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:15.710281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.710281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:15.710281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:15.710281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:15.710281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:15.710281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.710281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:15.710848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:15.711248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.711248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:15.711248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.711248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.711665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:15.711990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:15.712313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.712313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:15.712313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.712313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:15.722007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:15.722007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:15.722007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:15.722007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:15.722296 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.722445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:15.722697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.724036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:15.724036 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.724036 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:15.724036 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:15.724036 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.724036 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:15.724036 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.724036 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:15.725042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.725042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:15.725042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:15.725042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:15.725042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.725042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:15.725042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:15.725042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.725042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:15.725042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:15.725042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:15.725042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.725374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:15.725631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.725631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:15.726213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:15.726531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:15.727012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:15.727012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:15.727012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:19.138247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:19.140554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:19.140554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:19.140554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:19.155661 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=233 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 00:59:19.186717 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:19.186867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:19.189231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:19.189706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:19.190107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:19.190511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:19.191104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:19.191104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:19.191104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:19.191104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:19.191104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:19.191104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:19.191104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:19.191104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:19.191104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:20.631752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:20.632501 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:20.632501 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:20.634003 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.635060 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.635361 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ab85ec31-06e8-4c65-a2de-6ae5d5a75dd7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.635516 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.635634 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.635744 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.635848 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.635947 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.636061 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-9161f55f-20ce-47c8-8b81-d370945a4dce, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.636163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:20.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:20.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:20.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:20.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.636772 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:20.638009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.638009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:20.638009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:20.638009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:20.638009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.638009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:20.638009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:20.638009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.638009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:20.638009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:20.638009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:20.638009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.638356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:20.638655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.638655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:20.639052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:20.639615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.640040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:20.640040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.640040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:20.678911 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:20.680689 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:20.680689 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:20.680689 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:59:20.680689 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.680689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.681673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:20.681673 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.681673 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:20.681673 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:20.681673 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.681673 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:20.681673 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.681673 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:20.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:20.682401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:20.682401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:20.682401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.682401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:20.682401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:20.682401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.682401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:20.682401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.682401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:20.683375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.683375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:20.683375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:20.683375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:20.683375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.683375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:20.683375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:20.683375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:59:20.683375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:20.683375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:20.683375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.683375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:20.683733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:20.684037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.684037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:20.684037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.684037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:20.684219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:20.685120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:20.686370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.686370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:20.686370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:20.686370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.686370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:20.686370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.686370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:20.707562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:20.707885 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:20.708030 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:20.708169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:20.708565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.708833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.709353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:20.709353 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.709353 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:20.709353 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:20.709353 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.709353 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:20.709353 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.709353 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:20.709961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:20.710313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:20.710313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:20.710313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.710313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:20.710313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:20.710313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.710313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:20.710313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.710313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:20.710706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.710706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:20.710706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:20.710706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:20.710706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.710706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:20.710706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:20.710706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:20.710706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:20.710706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.710706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:20.711129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:20.711568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.711568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:20.711568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.711568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.711646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:20.712044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:20.712377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.712377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:20.712377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.712377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:20.721611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:20.721846 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:20.722014 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:20.722180 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:20.722497 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.722726 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:20.722990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.723623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:20.723623 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.723623 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:20.723623 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:20.723623 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.723623 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:20.723623 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.723623 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:20.724531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.724531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:20.724531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:20.724531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:20.724531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.724531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:20.724531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:20.724531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.724531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:20.724531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:20.724531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:20.724531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.724926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:20.725490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.725490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:20.725563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:20.726026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:20.726473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:20.726473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:20.726473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:24.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:24.139944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:24.139944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:24.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:24.190576 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:24.190576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:24.190806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:24.190806 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:24.190806 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:24.190806 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:24.190806 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:24.190806 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:24.190806 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:24.190806 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:24.191592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:24.191592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:24.191592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:24.191592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:24.191592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:24.191592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:24.191592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:24.191592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:24.191592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:24.191592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:24.191592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:24.191592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:24.191910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:24.192331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:24.192694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:24.193086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:24.193086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:24.193086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:24.193086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:24.193086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:24.193086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:25.632214 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:25.632617 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:25.632711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:25.633563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:25.633810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:25.634003 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:25.634174 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:25.634357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:25.634517 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:25.634672 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:25.634781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.635391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:25.635391 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.635391 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:25.635391 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:25.635391 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.635391 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:25.635391 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.635391 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:25.636089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.636089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:25.636089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:25.636089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:25.636089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.636089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:25.636089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:25.636089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.636089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:25.636089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:25.636089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:25.636089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.636516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:25.636924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:25.637543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.638176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:25.638176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:25.638176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.638176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:25.638176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.638176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:25.681348 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:25.681731 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:25.681824 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:25.681930 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:59:25.682505 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:25.682720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.684060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:25.684060 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.684060 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:25.684060 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:25.684060 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.684060 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:25.684060 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.684060 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:25.684654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:25.684980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:25.684980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:25.684980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.684980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:25.684980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:25.684980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.684980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:25.684980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.684980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:25.685373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.685373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:25.685373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:25.685373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:25.685373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.685373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:25.685373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:25.685373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:59:25.685373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:25.685373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:25.685373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.685373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:25.685649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:25.685907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.685907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:25.685907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.685907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:25.686087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:25.686416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:25.686752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.686752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:25.686752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:25.686752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.686752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:25.686752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.686752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:25.708019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:25.708223 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:25.708314 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:25.708417 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:25.708916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:25.709232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.710410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:25.710410 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.710410 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:25.710410 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:25.710410 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.710410 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:25.710410 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.710410 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:25.711208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:25.711685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:25.711685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:25.711685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.711685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:25.711685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:25.711685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.711685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:25.711685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.711685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:25.712917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.712917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:25.712917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:25.712917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:25.712917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.712917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:25.712917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:25.712917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:25.712917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:25.712917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.712917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:25.713347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:25.713768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.713768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:25.713768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.713768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.714978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:25.715546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:25.716641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.716641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:25.716641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.716641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:25.721883 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:25.722116 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:25.722211 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:25.722354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:25.722768 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:25.723011 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:25.723206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.723670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:25.723670 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.723670 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:25.723670 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:25.723670 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.723670 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:25.723670 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.723670 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:25.724616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.724616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:25.724616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:25.724616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:25.724616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.724616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:25.724616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:25.724616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.724616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:25.724616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:25.724616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:25.724616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.725049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:25.725494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.725494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:25.726158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:25.726729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:25.727386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:25.727386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:25.727386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:29.139841 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:29.140276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:29.140313 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:29.141014 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:29.190274 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:29.190887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:29.191739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:29.191739 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:29.191739 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:29.191739 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:29.191739 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:29.191739 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:29.191739 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:29.191739 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:29.192469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:29.192469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:29.192469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:29.192469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:29.192469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:29.192469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:29.192469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:29.192469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:29.192469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:29.192469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:29.192469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:29.192469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:29.192885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:29.193283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:29.193728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:29.194333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:29.194333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:29.194333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:29.194333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:29.194333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:29.194333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:30.632700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:30.633152 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:30.633152 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:30.634091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.634254 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-077857fc-a6a2-4b15-843e-55a6f95312d4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.634399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.634755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.634755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.634755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.634885 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.635039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.635160 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.635348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.635727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:30.635727 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.635727 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:30.635727 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:30.635727 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.635727 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:30.635727 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.635727 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:30.636499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.636499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:30.636499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:30.636499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:30.636499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.636499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:30.636499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:30.636499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.636499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:30.636499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:30.636499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:30.636499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.636787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:30.637079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:30.637445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.637778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:30.637778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:30.637778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.637778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:30.637778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.637778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:30.681906 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:30.682406 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:30.682511 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:30.682656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:59:30.683038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.683247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.683681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:30.683681 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.683681 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:30.683681 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:30.683681 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.683681 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:30.683681 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.683681 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:30.684535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:30.685004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:30.685004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:30.685004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.685004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:30.685004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:30.685004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.685004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:30.685004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.685004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:30.685392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.685392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:30.685392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:30.685392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:30.685392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.685392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:30.685392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:30.685392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:59:30.685392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:30.685392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:30.685392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.685392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:30.685770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:30.686173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:30.686648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:30.687176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:30.687176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:30.687176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:30.687176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:30.687176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:30.687176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.687176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:30.687176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:30.687176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.687176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:30.687176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.687176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:30.708481 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:30.710352 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:30.710908 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:30.710908 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:30.711778 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.712120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.713351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:30.713351 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.713351 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:30.713351 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:30.713351 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.713351 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:30.713351 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.713351 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:30.714304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:30.714597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:30.714597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:30.714597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.714597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:30.714597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:30.714597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.714597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:30.714597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.714597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:30.715255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.715255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:30.715255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:30.715255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:30.715255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.715255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:30.715255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:30.715255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:30.715255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:30.715255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.715255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:30.715490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:30.715733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.715733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:30.715733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.715733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.717041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:30.717418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:30.717745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.717745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:30.717745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.717745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:30.722482 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:30.723109 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:30.723258 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:30.723438 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:30.724448 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.724448 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:30.725073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.725760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:30.725760 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.725760 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:30.725760 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:30.725760 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.725760 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:30.725760 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.725760 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:30.728588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.728588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:30.728588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:30.728588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:30.728588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.728588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:30.728588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:30.728588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.728588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:30.728588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:30.728588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:30.728588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.729054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:30.729476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.729476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:30.732099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:30.732618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:30.733434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:30.733434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:30.733434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:34.140344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:34.140779 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:34.140974 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:34.141943 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:34.185595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:34.185739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:34.186458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:34.186458 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:34.186458 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:34.186458 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:34.186458 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:34.186458 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:34.186458 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:34.186458 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:34.188370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:34.188370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:34.188370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:34.188370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:34.188370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:34.188370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:34.188370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:34.188370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:34.188370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:34.188370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:34.188370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:34.188370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:34.189132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:34.189535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:34.189876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:34.190347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:34.190347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:34.190347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:34.190347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:34.190347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:34.190347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:35.633287 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:35.633815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:35.633815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:35.635494 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:35.635706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:35.635789 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:35.635917 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:35.636054 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:35.636176 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:35.636615 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:35.636777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:35.636910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.639303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:35.639303 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.639303 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:35.639303 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:35.639303 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.639303 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:35.639303 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.639303 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:35.639920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.639920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:35.639920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:35.639920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:35.639920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.639920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:35.639920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:35.639920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.639920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:35.639920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:35.639920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:35.639920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.640166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:35.640778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:35.641037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.641356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:35.641356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:35.641356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.641356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:35.641356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.641356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:35.682578 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:35.683042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:35.683108 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:35.683215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:59:35.684215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:35.684215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.684308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:35.684308 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.684308 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:35.684308 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:35.684308 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.684308 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:35.684308 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.684308 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:35.684860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:35.685213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:35.685213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:35.685213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.685213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:35.685213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:35.685213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.685213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:35.685213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.685213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:35.685483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.685483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:35.685483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:35.685483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:35.685483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.685483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:35.685483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:35.685483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:59:35.685483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:35.685483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:35.685483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.685483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:35.685725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:35.685974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.685974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:35.685974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.685974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:35.686165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:35.686464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:35.686763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.686763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:35.686763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:35.686763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.686763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:35.686763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.686763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:35.710398 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:35.712097 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:35.712097 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:35.712097 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:35.712097 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:35.712097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.712312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:35.712312 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.712312 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:35.712312 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:35.712312 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.712312 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:35.712312 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.712312 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:35.712811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:35.713103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:35.713103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:35.713103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.713103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:35.713103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:35.713103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.713103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:35.713103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.713103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:35.713862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.713862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:35.713862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:35.713862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:35.713862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.713862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:35.713862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:35.713862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:35.713862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:35.713862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.713862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:35.714094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:35.714378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:35.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:35.714923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:35.714923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:35.714923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:35.714923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:35.714923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.714923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:35.714923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:35.714923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.714923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:35.714923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.714923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:35.723415 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:35.723642 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:35.723752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:35.723852 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:35.724257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:35.724470 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:35.724677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.725126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:35.725126 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.725126 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:35.725126 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:35.725126 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.725126 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:35.725126 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.725126 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:35.725744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.725744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:35.725744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:35.725744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:35.725744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.725744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:35.725744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:35.725744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.725744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:35.725744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:35.725744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:35.725744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.726005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:35.726249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.726249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:35.726463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:35.726791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:35.727226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:35.727226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:35.727226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:39.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:39.143797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:39.146138 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:39.146138 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:39.195664 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:39.195895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.052 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:39.198660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:39.199720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:39.200295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:39.200982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:39.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:39.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:39.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:39.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:39.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:39.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:39.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:39.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:39.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:40.633661 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:40.635272 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:40.635508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:40.637812 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.637921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.638057 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.638218 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.638367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.638490 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.638616 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-971866bd-7f65-482c-97a4-60acf79a80fb, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.638742 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.638883 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.639011 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.639135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:40.642526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:40.643070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.643482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:40.643932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:40.644536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:40.644536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:40.644536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:40.644536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.644536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:40.644536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:40.644536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.644536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:40.644536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.644536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:40.684645 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:40.685537 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:40.685537 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:40.685537 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:59:40.685891 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.686105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.689751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:40.690314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:40.690733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.691094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:40.691501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:40.691925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:40.692477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:40.692477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:40.692477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.692477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:40.692477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:40.692477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.692477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:40.692477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.692477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:40.710920 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:40.712421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:40.712703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:40.713122 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:40.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.714024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.714593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:40.714593 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.714593 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:40.714593 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:40.714593 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.714593 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:40.714593 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.714593 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:40.716485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:40.717452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:40.717452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:40.717452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.717452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:40.717452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:40.717452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.717452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:40.717452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.717452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:40.719461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.719461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:40.719461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:40.719461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:40.719461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.719461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:40.719461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:40.719461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:40.719461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:40.719461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.719461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:40.720136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:40.721265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.721265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:40.721265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.721265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.728474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:40.729079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:40.729711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.729711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:40.729711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.729711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:40.731261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:40.731261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:40.733771 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:40.733901 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:40.734348 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.736088 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:40.736088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.736440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:40.736440 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.736440 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:40.736440 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:40.736440 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.736440 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:40.736440 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.736440 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:40.737454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.737454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:40.737454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:40.737454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:40.737454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.737454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:40.737454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:40.737454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.737454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:40.737454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:40.737454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:40.737454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.737844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:40.738258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.738258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:40.738541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:40.739232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:40.739555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:40.739555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:40.739555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:44.143851 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:44.144809 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:44.144895 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:44.145573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:44.183922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:44.184511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:44.184805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:44.184805 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:44.184805 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:44.184805 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:44.184805 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:44.184805 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:44.184805 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:44.184805 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:44.185650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:44.185650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:44.185650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:44.185650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:44.185650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:44.185650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:44.185650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:44.185650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:44.185650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:44.185650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:44.185650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:44.185650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:44.186236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:44.186842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:44.187495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:44.189630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:44.189630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:44.189630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:44.189630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:44.189630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:44.189630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:45.634795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:45.635647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:45.635647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:45.636603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.636839 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.636985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7b80d822-209f-40fb-922d-3aab2eaa08bd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.637093 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d4aaf6ce-5920-48d0-b4ab-b1ca4a5933be, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.637576 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.637697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.637820 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.637920 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.638080 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.638232 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.638373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.638899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:45.638899 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.638899 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:45.638899 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:45.638899 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.638899 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:45.638899 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.638899 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:45.639718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.639718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:45.639718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:45.639718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:45.639718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.639718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:45.639718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:45.639718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.639718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:45.639718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:45.639718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:45.639718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.640140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:45.640524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.640524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:45.640587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:45.641010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.641531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:45.641531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.641531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:45.685105 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:45.686042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:45.686203 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:45.686369 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:59:45.687294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.687462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.688022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:45.688022 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.688022 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:45.688022 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:45.688022 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.688022 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:45.688022 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.688022 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:45.688968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:45.689301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:45.689301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:45.689301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.689301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:45.689301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:45.689301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.689301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:45.689301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.689301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:45.690478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.690478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:45.690478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:45.690478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:45.690478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.690478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:45.690478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:45.690478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:59:45.690478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:45.690478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:45.690478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.690478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:45.690792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:45.691204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.691204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:45.691204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.691204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:45.694856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:45.695283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:45.695636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.695636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:45.695636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:45.695636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.695636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:45.695636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.695636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:45.712443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:45.712946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:45.713128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:45.713305 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:45.713758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.714004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:45.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:45.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:45.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:45.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.714624 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:45.715933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:45.716315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:45.716315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:45.716315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.716315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:45.716315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:45.716315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.716315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:45.716315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.716315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:45.717622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.717622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:45.717622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:45.717622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:45.717622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.717622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:45.717622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:45.717622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:45.717622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:45.717622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.717622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:45.718030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:45.718321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.718321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:45.718321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.718321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.720044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:45.720401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:45.720749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.720749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:45.720749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.720749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:45.731100 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:45.731544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:45.731691 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:45.731874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:45.733127 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.733398 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:45.733646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.734183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:45.734183 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.734183 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:45.734183 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:45.734183 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.734183 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:45.734183 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.734183 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:45.735230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.735230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:45.735230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:45.735230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:45.735230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.735230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:45.735230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:45.735230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.735230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:45.735230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:45.735230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:45.735230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.735492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:45.735692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.735692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:45.736243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:45.736639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:45.736989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:45.736989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:45.736989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:49.148351 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:49.152683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:49.152683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:49.152683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:49.210368 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:49.210704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:49.211482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:49.211482 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:49.211482 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:49.211482 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:49.211482 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:49.211482 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:49.211482 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:49.211482 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:49.213065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.061 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:49.213065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:49.213065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:49.213065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:49.213065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:49.213065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:49.213065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:49.213065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:49.213065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:49.213065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:49.213065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:49.213065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:49.213496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:49.213860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:49.213860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:49.216934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:49.217384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:49.217739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:49.217739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:49.217739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:50.635337 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:50.635835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:50.635835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:50.637247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:50.637385 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:50.637711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:50.637711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:50.637807 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:50.637926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:50.638116 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:50.638284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:50.638430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:50.640514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:50.640895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.641218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:50.641463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:50.641826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:50.641826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:50.641826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:50.641826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.641826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:50.641826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:50.641826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.641826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:50.641826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.641826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:50.689023 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:50.689562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:50.689823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:50.690530 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:59:50.691057 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:50.691286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.694174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:50.694556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:50.694861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.695118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:50.695401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:50.695708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:50.696042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:50.696042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:50.696042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.696042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:50.696042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:50.696042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.696042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:50.696042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.696042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:50.713067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:50.714144 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:50.714144 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:50.714144 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:50.715647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:50.717453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.717801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:50.717801 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.717801 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:50.717801 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:50.717801 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.717801 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:50.717801 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.717801 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:50.720548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:50.721569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:50.721569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:50.721569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.721569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:50.721569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:50.721569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.721569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:50.721569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.721569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:50.724594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.724594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:50.724594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:50.724594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:50.724594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.724594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:50.724594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:50.724594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:50.724594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:50.724594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.724594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:50.724992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:50.725430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:50.725852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:50.727638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:50.727638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:50.727638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:50.727638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:50.727638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.727638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:50.727638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:50.727638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.727638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:50.727638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.727638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:50.731579 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:50.731850 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:50.732290 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:50.732290 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:50.733542 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:50.733842 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:50.735525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.736821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:50.736821 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.736821 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:50.736821 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:50.736821 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.736821 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:50.736821 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.736821 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:50.737710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.737710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:50.737710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:50.737710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:50.737710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.737710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:50.737710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:50.737710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.737710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:50.737710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:50.737710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:50.737710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.738766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:50.739831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:50.740451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:50.741012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:50.741012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:50.741012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:50.741012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:50.741012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:50.741012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:54.152300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:54.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:54.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:54.153520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:54.189602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:54.190063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:54.190539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:54.190539 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:54.190539 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:54.190539 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:54.190539 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:54.190539 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:54.190539 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:54.190539 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:54.191231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:54.191231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:54.191231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:54.191231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:54.191231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:54.191231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:54.191231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:54.191231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:54.191231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:54.191231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:54.191231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:54.191231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:54.191693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:54.192307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:54.192804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:54.193867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:54.193867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:54.193867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:54.193867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:54.193867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:54.193867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:55.635654 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:55.636157 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:55.636314 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:55.637407 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.637584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.637684 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.637797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.637895 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.638007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.638141 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.638281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.638392 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.638493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.638919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:55.638919 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.638919 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:55.638919 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:55.638919 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.638919 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:55.638919 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.638919 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:55.639851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.639851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:55.639851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:55.639851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:55.639851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.639851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:55.639851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:55.639851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.639851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:55.639851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:55.639851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:55.639851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.640243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:55.640484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.640484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:55.640983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:55.641346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.641706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:55.641706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.641706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:55.689554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:55.689822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:55.689929 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:55.690074 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 00:59:55.690385 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.690540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.690930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:55.690930 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.690930 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:55.690930 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:55.690930 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.690930 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:55.690930 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.690930 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:55.691577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:55.691990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:55.691990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:55.691990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.691990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:55.691990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:55.691990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.691990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:55.691990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.691990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:55.692705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.692705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:55.692705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:55.692705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:55.692705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.692705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:55.692705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:55.692705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 00:59:55.692705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:55.692705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:55.692705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.692705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:55.693005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:55.693265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.693265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:55.693265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.693265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:55.693885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:55.694526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:55.695166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.695166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:55.695166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:55.695166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.695166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:55.695166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.695166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:55.713831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:55.714199 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:55.714307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:55.714413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:55.714896 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.715207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.715817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:55.715817 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.715817 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:55.715817 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:55.715817 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.715817 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:55.715817 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.715817 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 00:59:55.716474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 00:59:55.716752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:55.716752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 00:59:55.716752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.716752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 00:59:55.716752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 00:59:55.716752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.716752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 00:59:55.716752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.716752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 00:59:55.718463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.718463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:55.718463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:55.718463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:55.718463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.718463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:55.718463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:55.718463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:55.718463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:55.718463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.718463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:55.718851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:55.719300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:55.719748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:55.720344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:55.720344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:55.720344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:55.720344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:55.720344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.720344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:55.720344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:55.720344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.720344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:55.720344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.720344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:55.731986 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:55.732340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:55.732426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:55.732558 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:55.734439 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.734614 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:55.734937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.736447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:55.736447 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.736447 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:55.736447 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:55.736447 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.736447 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:55.736447 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.736447 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:55.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:55.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:55.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:55.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:55.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:55.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:55.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:55.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:55.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.738034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:55.738404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.738404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:55.739154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:55.739695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:55.740326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:55.740326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:55.740326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 00:59:59.152839 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 00:59:59.153523 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 00:59:59.153523 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 00:59:59.154253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 00:59:59.230040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 00:59:59.230040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:59.230040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 00:59:59.230040 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:59.230040 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 00:59:59.230040 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 00:59:59.230040 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:59.230040 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 00:59:59.230040 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:59.230040 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 00:59:59.230685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.077 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:59.230685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 00:59:59.230685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:59.230685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 00:59:59.230685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:59.230685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:59.230685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:59.230685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:59.230685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:59.230685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:59.230685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 00:59:59.230685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:59.230939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 00:59:59.231199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:59.231199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 00:59:59.231357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 00:59:59.231658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 00:59:59.232028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 00:59:59.232028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 00:59:59.232028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:00.636126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:00.638675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:00.638763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:00.639812 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.640006 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.640138 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.640356 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.640434 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.640560 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.640688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.640804 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.640936 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.641208 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.641208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.641759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:00.641759 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.641759 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:00.641759 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:00.641759 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.641759 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:00.641759 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.641759 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:00.642367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.642367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:00.642367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:00.642367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:00.642367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.642367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:00.642367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:00.642367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.642367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:00.642367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:00.642367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:00.642367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.642623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:00.643036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.643036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:00.646207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:00.646561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.646937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:00.646937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.646937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:00.689990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:00.692491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:00.692576 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:00.692667 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:00:00.693046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.693271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.693770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:00.693770 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.693770 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:00.693770 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:00.693770 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.693770 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:00.693770 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.693770 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:00.694706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:00.694977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:00.694977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:00.694977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.694977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:00.694977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:00.694977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.694977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:00.694977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.694977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:00.695289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.695289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:00.695289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:00.695289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:00.695289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.695289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:00.695289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:00.695289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:00:00.695289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:00.695289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:00.695289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.695289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:00.695503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:00.695740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.695740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:00.695740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.695740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:00.696327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:00.696669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:00.697021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.697021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:00.697021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:00.697021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.697021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:00.697021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.697021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:00.714245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:00.714676 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:00.714777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:00.714929 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:00.717046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.717046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.717802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:00.717802 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.717802 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:00.717802 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:00.717802 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.717802 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:00.717802 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.717802 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:00.718671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:00.719036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:00.719036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:00.719036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.719036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:00.719036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:00.719036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.719036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:00.719036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.719036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:00.719822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.719822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:00.719822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:00.719822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:00.719822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.719822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:00.719822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:00.719822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:00.719822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:00.719822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.719822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:00.720134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:00.720409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.720409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:00.720409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.720409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.720880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:00.721225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:00.721547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.721547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:00.721547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.721547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:00.732401 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:00.732805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:00.732937 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:00.733159 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:00.733520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.733774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:00.734145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.734963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:00.734963 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.734963 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:00.734963 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:00.734963 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.734963 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:00.734963 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.734963 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:00.737245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.737245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:00.737245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:00.737245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:00.737245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.737245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:00.737245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:00.737245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.737245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:00.737245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:00.737245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:00.737245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.737508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:00.737797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.737797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:00.738558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:00.738885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:00.739710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:00.739710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:00.739710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:04.153484 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:04.155021 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:04.155173 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:04.156281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:04.211255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:04.211612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:04.212104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:04.212104 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:04.212104 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:04.212104 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:04.212104 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:04.212104 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:04.212104 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:04.212104 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:04.217429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:04.217429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:04.217429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:04.217429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:04.217429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:04.217429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:04.217429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:04.217429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:04.217429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:04.217429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:04.217429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:04.217429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:04.217828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:04.218115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:04.218115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:04.219087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:04.220079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:04.220620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:04.220620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:04.220620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:05.638675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:05.639123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:05.639123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:05.640077 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.640282 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.640408 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.640526 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.640627 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.640727 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.640866 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.640971 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.641068 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.641163 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.641278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d605d268-7264-4349-9633-1e26c9359915, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.641381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.641834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:05.641834 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.641834 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:05.641834 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:05.641834 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.641834 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:05.641834 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.641834 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:05.642634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.642634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:05.642634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:05.642634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:05.642634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.642634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:05.642634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:05.642634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.642634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:05.642634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:05.642634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:05.642634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.642903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:05.643148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.643148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:05.643789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:05.644133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.644563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:05.644563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.644563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:05.692559 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:05.692928 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:05.693030 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:05.693808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:00:05.693808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.693808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.694310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:05.694310 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.694310 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:05.694310 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:05.694310 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.694310 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:05.694310 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.694310 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:05.694920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:05.695292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:05.695292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:05.695292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.695292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:05.695292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:05.695292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.695292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:05.695292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.695292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:05.695573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.695573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:05.695573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:05.695573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:05.695573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.695573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:05.695573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:05.695573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:00:05.695573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:05.695573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:05.695573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.695573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:05.695827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:05.696220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.696220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:05.696220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.696220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:05.696469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:05.696836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:05.697183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.697183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:05.697183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:05.697183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.697183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:05.697183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.697183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:05.714808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:05.715109 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:05.715214 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:05.715353 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:05.715768 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.715980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.716432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:05.716432 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.716432 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:05.716432 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:05.716432 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.716432 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:05.716432 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.716432 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:05.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:05.717380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:05.717380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:05.717380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.717380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:05.717380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:05.717380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.717380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:05.717380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.717380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:05.717913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.717913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:05.717913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:05.717913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:05.717913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.717913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:05.717913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:05.717913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:05.717913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:05.717913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.717913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:05.718171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:05.718449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:05.718702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:05.719089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:05.719089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:05.719089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:05.719089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:05.719089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.719089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:05.719089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:05.719089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.719089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:05.719089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.719089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:05.732809 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:05.733177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:05.733302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:05.733436 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:05.735474 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.735671 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:05.735962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.736398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:05.736398 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.736398 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:05.736398 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:05.736398 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.736398 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:05.736398 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.736398 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:05.737032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.737032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:05.737032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:05.737032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:05.737032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.737032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:05.737032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:05.737032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.737032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:05.737032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:05.737032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:05.737032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.737337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:05.737557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.737557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:05.737870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:05.738273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:05.738615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:05.738615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:05.738615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:09.155057 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:09.155575 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:09.155575 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:09.156009 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:09.198258 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:09.199460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:09.200051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:09.200051 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:09.200051 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:09.200051 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:09.200051 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:09.200051 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:09.200051 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:09.200051 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:09.201032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:09.201032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:09.201032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:09.201032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:09.201032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:09.201032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:09.201032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:09.201032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:09.201032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:09.201032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:09.201032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:09.201032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:09.201367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:09.201621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:09.201621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:09.202175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:09.202573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:09.202955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:09.202955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:09.202955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:10.639205 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:10.639625 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:10.639716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:10.643404 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.643578 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.643704 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.643830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.643938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.644055 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.644152 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.644339 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.644476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.644622 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.644776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.645449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:10.645449 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.645449 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:10.645449 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:10.645449 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.645449 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:10.645449 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.645449 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:10.646246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.646246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:10.646246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:10.646246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:10.646246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.646246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:10.646246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:10.646246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.646246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:10.646246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:10.646246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:10.646246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.646652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:10.647096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:10.647560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.648124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:10.648124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:10.648124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.648124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:10.648124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.648124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:10.692938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:10.693498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:10.693734 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:10.693734 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:00:10.694132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.694324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.694974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:10.694974 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.694974 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:10.694974 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:10.694974 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.694974 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:10.694974 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.694974 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:10.695734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:10.696235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:10.696235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:10.696235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.696235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:10.696235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:10.696235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.696235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:10.696235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.696235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:10.696636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.696636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:10.696636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:10.696636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:10.696636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.696636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:10.696636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:10.696636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:00:10.696636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:10.696636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:10.696636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.696636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:10.697025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:10.697578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:10.697871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:10.698241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:10.698241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:10.698241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:10.698241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:10.698241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:10.698241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.698241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:10.698241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:10.698241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.698241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:10.698241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.698241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:10.715627 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:10.715627 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:10.715627 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:10.715799 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:10.716809 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.717036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.717522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:10.717522 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.717522 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:10.717522 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:10.717522 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.717522 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:10.717522 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.717522 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:10.718138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:10.718452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:10.718452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:10.718452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.718452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:10.718452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:10.718452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.718452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:10.718452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.718452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:10.719109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.719109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:10.719109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:10.719109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:10.719109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.719109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:10.719109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:10.719109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:10.719109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:10.719109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.719109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:10.719419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:10.719747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.719747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:10.719747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.719747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.720209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:10.720602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:10.721793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.721793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:10.721793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.721793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:10.733209 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:10.733830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:10.733830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:10.733830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:10.735642 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.735906 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:10.736304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:10.739176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.739471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:10.739772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:10.740232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:10.740232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:10.740232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:10.740232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:10.740232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:10.740232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:10.740232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:10.740232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:10.740232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:10.740232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:14.155507 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:14.157665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:14.157665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:14.157665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:14.238146 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:14.238328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:14.238700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:14.238700 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:14.238700 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:14.238700 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:14.238700 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:14.238700 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:14.238700 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:14.238700 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:14.239671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.082 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:14.239671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:14.239671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:14.239671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:14.239671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:14.239671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:14.239671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:14.239671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:14.239671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:14.239671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:14.239671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:14.239671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:14.239970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:14.240288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:14.240288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:14.240677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:14.241060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:14.241524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:14.241524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:14.241524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:15.642531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:15.643031 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:15.643168 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:15.644515 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.644706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.644848 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.645008 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.645149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.645300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.645427 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.645566 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.645696 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.645818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.645962 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.646099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.646718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:15.646718 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.646718 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:15.646718 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:15.646718 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.646718 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:15.646718 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.646718 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:15.647691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.647691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:15.647691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:15.647691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:15.647691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.647691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:15.647691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:15.647691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.647691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:15.647691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:15.647691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:15.647691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.648329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:15.648728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.648728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:15.649905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:15.650479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.651097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:15.651097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.651097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:15.696884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:15.697888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:15.697888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:15.697888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:00:15.698334 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.698529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.699651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:15.699651 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.699651 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:15.699651 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:15.699651 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.699651 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:15.699651 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.699651 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:15.700315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:15.700862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:15.700862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:15.700862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.700862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:15.700862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:15.700862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.700862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:15.700862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.700862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:15.701183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.701183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:15.701183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:15.701183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:15.701183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.701183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:15.701183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:15.701183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:00:15.701183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:15.701183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:15.701183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.701183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:15.701586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:15.703118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:15.703667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:15.704100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:15.704100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:15.704100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:15.704100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:15.704100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:15.704100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.704100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:15.704100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:15.704100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.704100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:15.704100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.704100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:15.715602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:15.716795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:15.716906 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:15.717066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:15.717530 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.717708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.718256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:15.718256 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.718256 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:15.718256 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:15.718256 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.718256 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:15.718256 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.718256 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:15.720227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:15.720623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:15.720623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:15.720623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.720623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:15.720623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:15.720623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.720623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:15.720623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.720623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:15.720820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.720820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:15.720820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:15.720820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:15.720820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.720820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:15.720820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:15.720820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:15.720820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:15.720820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.720820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:15.721236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:15.721801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.721801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:15.721801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.721801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.724010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:15.724489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:15.725224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.725224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:15.725224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.725224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:15.733643 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:15.734108 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:15.734747 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:15.734747 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:15.735068 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.735327 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:15.735690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.736618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:15.736618 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.736618 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:15.736618 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:15.736618 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.736618 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:15.736618 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.736618 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:15.737909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.737909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:15.737909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:15.737909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:15.737909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.737909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:15.737909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:15.737909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.737909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:15.737909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:15.737909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:15.737909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.738232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:15.738483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.738483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:15.739026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:15.739827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:15.740270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:15.740270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:15.740270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:19.156445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:19.158070 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:19.158231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:19.159872 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:19.167169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=238 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:00:19.217970 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.061 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:19.220598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:19.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:19.221609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:19.221609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:19.221609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:19.221609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:19.221609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:19.221609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:19.221609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:19.224955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:19.225414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:19.225965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:19.225965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:19.225965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:20.643800 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:20.645762 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:20.646387 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:20.648273 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.648430 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.648541 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.648652 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.648750 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.648853 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.648976 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.649063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.649159 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.649279 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.649379 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.649474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.651419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:20.651419 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.651419 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:20.651419 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:20.651419 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.651419 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:20.651419 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.651419 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:20.652386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.652386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:20.652386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:20.652386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:20.652386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.652386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:20.652386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:20.652386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.652386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:20.652386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:20.652386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:20.652386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.652652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:20.652899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.652899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:20.656378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:20.656874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.657444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:20.657444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.657444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:20.698445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:20.699738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:20.699874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:20.700027 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:00:20.700476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.700659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.701212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:20.701212 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.701212 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:20.701212 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:20.701212 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.701212 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:20.701212 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.701212 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:20.702141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:20.702574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:20.702574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:20.702574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.702574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:20.702574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:20.702574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.702574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:20.702574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.702574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:20.704451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.704451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:20.704451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:20.704451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:20.704451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.704451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:20.704451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:20.704451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:00:20.704451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:20.704451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:20.704451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.704451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:20.704783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:20.705146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.705146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:20.705146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.705146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:20.711861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:20.712378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:20.712804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.712804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:20.712804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:20.712804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.712804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:20.712804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.712804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:20.718285 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:20.718891 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:20.719066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:20.719406 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:20.721556 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.721786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.724015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.724660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:20.726241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.726241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:20.726241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:20.726241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:20.726241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.726241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:20.726241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:20.726241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:20.726241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:20.726241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.726241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:20.726579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:20.726919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.726919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:20.726919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.726919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.728979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:20.729563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:20.730085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.730085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:20.730085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.730085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:20.734335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:20.734901 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:20.735170 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:20.736069 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:20.736069 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.736262 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:20.736637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.737324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:20.737324 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.737324 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:20.737324 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:20.737324 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.737324 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:20.737324 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.737324 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:20.739007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.739007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:20.739007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:20.739007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:20.739007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.739007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:20.739007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:20.739007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.739007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:20.739007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:20.739007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:20.739007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.739378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:20.739967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:20.740352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:20.741108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:20.741108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:20.741108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:20.741108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:20.741108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:20.741108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:24.158445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:24.159401 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:24.159401 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:24.160262 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:24.252682 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:24.252915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:24.253569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:24.253569 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:24.253569 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:24.253569 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:24.253569 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:24.253569 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:24.253569 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:24.253569 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:24.254546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.094 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:24.254546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:24.254546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:24.254546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:24.254546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:24.254546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:24.254546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:24.254546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:24.254546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:24.254546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:24.254546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:24.254546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:24.254888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:24.255240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:24.255240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:24.256221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:24.256626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:24.257061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:24.257061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:24.257061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:25.646514 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:25.647092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:25.647092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:25.652238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.652238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.652238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.652238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.652238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.652761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.652761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.652761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.652761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.652761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.653458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.653458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.653736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:25.653736 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.653736 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:25.653736 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:25.653736 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.653736 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:25.653736 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.653736 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:25.655026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.655026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:25.655026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:25.655026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:25.655026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.655026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:25.655026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:25.655026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.655026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:25.655026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:25.655026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:25.655026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.657146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:25.657545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.657545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:25.659237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:25.659783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.660203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:25.660203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.660203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:25.699729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:25.700598 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:25.700598 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:25.700598 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:00:25.702208 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.702208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.705166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:25.705166 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.705166 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:25.705166 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:25.705166 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.705166 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:25.705166 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.705166 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:25.706365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:25.706732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:25.706732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:25.706732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.706732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:25.706732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:25.706732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.706732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:25.706732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.706732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:25.709668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.709668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:25.709668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:25.709668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:25.709668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.709668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:25.709668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:25.709668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:00:25.709668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:25.709668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:25.709668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.709668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:25.710043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:25.710405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.710405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:25.710405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.710405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:25.712365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:25.712792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:25.713273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.713273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:25.713273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:25.713273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.713273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:25.713273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.713273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:25.720773 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:25.721859 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:25.721859 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:25.723082 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:25.724961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.725231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.725706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:25.725706 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.725706 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:25.725706 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:25.725706 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.725706 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:25.725706 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.725706 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:25.726865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:25.727819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:25.727819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:25.727819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.727819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:25.727819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:25.727819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.727819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:25.727819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.727819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:25.728342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.728342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:25.728342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:25.728342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:25.728342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.728342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:25.728342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:25.728342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:25.728342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:25.728342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.728342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:25.728609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:25.728884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.728884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:25.728884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.728884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.730247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:25.730709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:25.731202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.731202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:25.731202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.731202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:25.734852 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:25.735376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:25.735564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:25.735823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:25.736225 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.736464 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:25.739396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.740461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:25.740461 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.740461 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:25.740461 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:25.740461 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.740461 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:25.740461 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.740461 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:25.744790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.744790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:25.744790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:25.744790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:25.744790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.744790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:25.744790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:25.744790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.744790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:25.744790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:25.744790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:25.744790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.745546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:25.745893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:25.746390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:25.746873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:25.746873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:25.746873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:25.746873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:25.746873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:25.746873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:29.159238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:29.159802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:29.159842 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:29.160753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:29.241857 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:29.242012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:29.242527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:29.242527 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:29.242527 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:29.242527 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:29.242527 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:29.242527 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:29.242527 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:29.242527 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:29.243498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.083 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:29.243498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:29.243498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:29.243498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:29.243498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:29.243498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:29.243498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:29.243498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:29.243498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:29.243498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:29.243498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:29.243498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:29.244068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:29.244381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:29.244381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:29.244525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:29.245045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:29.245473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:29.245473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:29.245473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:30.646998 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:30.648635 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:30.648635 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:30.650047 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.650047 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.650047 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.650171 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.650309 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.650430 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.650574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.650715 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.650856 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.651068 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.651232 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.651419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.652281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:30.652281 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.652281 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:30.652281 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:30.652281 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.652281 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:30.652281 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.652281 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:30.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:30.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:30.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:30.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:30.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:30.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:30.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:30.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:30.653342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.653588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:30.655474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:30.655782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.656094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:30.656094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:30.656094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.656094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:30.656094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.656094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:30.700492 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:30.700492 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:30.701341 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:30.701341 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:30.703506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.703928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:30.704394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.704394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:30.704394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.704394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:30.704394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:30.704394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:30.704394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.704394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:30.704394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:30.704394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:00:30.704394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:30.704394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:30.704759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.705111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.710882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.713733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:30.721725 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:30.722103 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:30.722206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:30.722360 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:30.722747 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.722973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.724091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.724413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:30.724646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.724646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:30.724646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:30.724646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:30.724646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.724646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:30.724646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:30.724646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:30.724646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:30.724646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.724646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:30.724819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:30.725041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.725041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:30.725041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.725041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.728732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:30.729097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:30.729457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.729457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:30.729457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.729457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:30.735274 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:30.735614 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:30.735704 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:30.735841 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:30.736343 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.736525 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:30.736762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.739700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:30.739700 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.739700 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:30.739700 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:30.739700 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.739700 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:30.739700 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.739700 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:30.740480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.740480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:30.740480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:30.740480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:30.740480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.740480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:30.740480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:30.740480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.740480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:30.740480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:30.740480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:30.740480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.740731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:30.740917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.740917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:30.741202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:30.741461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:30.741737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:30.741737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:30.741737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:34.160786 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:34.161647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:34.161797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:34.162687 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:34.211255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:34.211486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:34.214919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:34.214919 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:34.214919 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:34.214919 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:34.214919 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:34.214919 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:34.214919 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:34.214919 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:34.218694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:34.218694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:34.218694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:34.218694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:34.218694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:34.218694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:34.218694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:34.218694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:34.218694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:34.218694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:34.218694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:34.218694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:34.219122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:34.219530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:34.219530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:34.222449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:34.222995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:34.223326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:34.223326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:34.223326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:35.648066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:35.648651 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:35.648651 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:35.649975 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.650645 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.650645 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.650645 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.650762 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.650875 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.651026 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.651157 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.651315 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.651458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.651588 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.651730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.652363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:35.652363 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.652363 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:35.652363 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:35.652363 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.652363 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:35.652363 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.652363 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:35.653204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.653204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:35.653204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:35.653204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:35.653204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.653204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:35.653204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:35.653204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.653204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:35.653204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:35.653204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:35.653204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.653534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:35.653849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.653849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:35.653982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:35.654411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.656457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:35.656457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.656457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:35.701917 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:35.701917 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:35.701917 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:35.701917 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:00:35.702224 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.702487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.707050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:35.707700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:35.708073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.708301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:35.708508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:35.708739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:35.709052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:35.709052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:35.709052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.709052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:35.709052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:35.709052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.709052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:35.709052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.709052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:35.723092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:35.723092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:35.723092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:35.723092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:35.723343 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.723568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.724484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:35.724484 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.724484 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:35.724484 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:35.724484 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.724484 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:35.724484 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.724484 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:35.725414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:35.725695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:35.725695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:35.725695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.725695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:35.725695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:35.725695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.725695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:35.725695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.725695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:35.726542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.726542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:35.726542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:35.726542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:35.726542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.726542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:35.726542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:35.726542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:35.726542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:35.726542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.726542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:35.726760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:35.726981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.726981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:35.726981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.726981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.728061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:35.728478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:35.728905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.728905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:35.728905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.728905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:35.735713 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:35.736067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:35.736496 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:35.736496 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:35.736715 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.736947 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:35.737183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:35.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:35.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:35.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:35.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.737617 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:35.738804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.738804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:35.738804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:35.738804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:35.738804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.738804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:35.738804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:35.738804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.738804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:35.738804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:35.738804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:35.738804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.739133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:35.739357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.739357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:35.739883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:35.740170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:35.740735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:35.740735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:35.740735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:39.162829 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:39.162829 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:39.162829 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:39.162829 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:39.204140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:39.204348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:39.204770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:39.204770 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:39.204770 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:39.204770 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:39.204770 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:39.204770 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:39.204770 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:39.204770 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:39.205481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:39.205481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:39.205481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:39.205481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:39.205481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:39.205481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:39.205481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:39.205481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:39.205481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:39.205481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:39.205481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:39.205481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:39.205865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:39.206262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:39.206712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:39.207259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:39.207259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:39.207259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:39.207259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:39.207259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:39.207259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:40.648520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:40.649101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:40.649101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:40.652554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.652833 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e1519e96-d739-48b1-a3f5-6cca2abd8544, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.653012 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.653216 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.653400 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.653575 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.653775 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.653923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.654108 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.654296 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.654469 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.654642 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.654813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.658898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:40.658898 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.658898 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:40.658898 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:40.658898 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.658898 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:40.658898 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.658898 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:40.660051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.660051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:40.660051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:40.660051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:40.660051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.660051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:40.660051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:40.660051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.660051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:40.660051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:40.660051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:40.660051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.660436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:40.660744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.660744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:40.661202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:40.661639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.662124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:40.662124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.662124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:40.701544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:40.701951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:40.702178 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:40.702375 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:00:40.702807 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.703081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.703707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:40.703707 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.703707 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:40.703707 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:40.703707 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.703707 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:40.703707 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.703707 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:40.704590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:40.705081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:40.705081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:40.705081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.705081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:40.705081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:40.705081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.705081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:40.705081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.705081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:40.705483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.705483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:40.705483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:40.705483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:40.705483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.705483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:40.705483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:40.705483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:00:40.705483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:40.705483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:40.705483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.705483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:40.705890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:40.706297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.706297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:40.706297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.706297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:40.706614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:40.707235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:40.707718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.707718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:40.707718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:40.707718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.707718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:40.707718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.707718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:40.722741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:40.723182 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:40.723403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:40.723563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:40.723962 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.724209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.724802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:40.724802 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.724802 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:40.724802 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:40.724802 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.724802 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:40.724802 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.724802 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:40.725841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:40.726355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:40.726355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:40.726355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.726355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:40.726355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:40.726355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.726355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:40.726355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.726355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:40.727010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.727010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:40.727010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:40.727010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:40.727010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.727010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:40.727010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:40.727010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:40.727010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:40.727010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.727010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:40.727349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:40.727677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.727677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:40.727677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.727677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.728903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:40.729532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:40.730274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.730274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:40.730274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.730274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:40.736138 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:40.736630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:40.737008 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:40.737439 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:40.737905 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.738208 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:40.738553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.739684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:40.739684 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.739684 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:40.739684 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:40.739684 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.739684 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:40.739684 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.739684 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:40.742170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.742170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:40.742170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:40.742170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:40.742170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.742170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:40.742170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:40.742170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.742170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:40.742170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:40.742170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:40.742170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.742594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:40.742899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.742899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:40.743456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:40.743908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:40.744404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:40.744404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:40.744404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:44.162158 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:44.163853 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:44.163853 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:44.164240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:44.246140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:44.246389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.084 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:44.251214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:44.251572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:44.251806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:44.252143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:44.252649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:44.252649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:44.252649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:44.252649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:44.252649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:44.252649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:44.252649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:44.252649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:44.252649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:45.649729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:45.650251 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:45.650251 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:45.651368 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.651520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.651677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-07f0d37d-ae4c-4d7b-a464-fbe7827abe92, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.651899 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.651982 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.652138 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.652304 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.652447 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.652591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.652734 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.652873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.653032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:45.654592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:45.655084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.655473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:45.655886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:45.656404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:45.656404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:45.656404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:45.656404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.656404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:45.656404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:45.656404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.656404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:45.656404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.656404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:45.702662 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:45.702662 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:45.702662 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:45.702662 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:00:45.703111 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.703363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.705409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:45.705409 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.705409 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:45.705409 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:45.705409 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.705409 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:45.705409 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.705409 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:45.706069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:45.706491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:45.706491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:45.706491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.706491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:45.706491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:45.706491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.706491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:45.706491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.706491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:45.709852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.709852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:45.709852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:45.709852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:45.709852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.709852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:45.709852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:45.709852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:00:45.709852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:45.709852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:45.709852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.709852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:45.710532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:45.710975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:45.712010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:45.712561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:45.712561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:45.712561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:45.712561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:45.712561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:45.712561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.712561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:45.712561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:45.712561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.712561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:45.712561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.712561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:45.724352 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:45.724756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:45.724877 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:45.725003 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:45.725509 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.725811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.726380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:45.726380 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.726380 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:45.726380 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:45.726380 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.726380 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:45.726380 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.726380 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:45.727113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:45.727594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:45.727594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:45.727594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.727594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:45.727594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:45.727594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.727594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:45.727594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.727594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:45.729675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.729675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:45.729675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:45.729675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:45.729675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.729675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:45.729675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:45.729675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:45.729675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:45.729675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.729675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:45.730053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:45.730447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.730447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:45.730447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.730447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.731438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:45.731901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:45.732409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.732409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:45.732409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.732409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:45.736689 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:45.737102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:45.737302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:45.737376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:45.737811 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.738087 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:45.738514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:45.740458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:45.740949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.741398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:45.741842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:45.742442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:45.742442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:45.742442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:45.742442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:45.742442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:45.742442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:45.742442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:45.742442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:45.742442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:45.742442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:49.163587 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:49.164243 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:49.164243 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:49.164590 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:49.215236 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:49.215236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:49.216056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:49.216056 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:49.216056 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:49.216056 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:49.216056 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:49.216056 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:49.216056 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:49.216056 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:49.217139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.052 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:49.217139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:49.217139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:49.217139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:49.217139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:49.217139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:49.217139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:49.217139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:49.217139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:49.217139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:49.217139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:49.217139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:49.217811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:49.218424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:49.218424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:49.218599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:49.219423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:49.221071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:49.221071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:49.221071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:50.650650 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:50.651172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:50.651172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:50.652578 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.652814 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.652982 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.653125 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.653297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.654330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.654330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.654414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.654567 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.654717 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.654864 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.655025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.657152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:50.657152 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.657152 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:50.657152 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:50.657152 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.657152 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:50.657152 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.657152 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:50.658122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.658122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:50.658122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:50.658122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:50.658122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.658122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:50.658122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:50.658122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.658122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:50.658122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:50.658122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:50.658122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.658540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:50.658885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.658885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:50.659242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:50.659689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.660081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:50.660081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.660081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:50.702653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:50.703625 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:50.703625 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:50.703625 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:00:50.703866 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.704137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.705425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:50.705425 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.705425 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:50.705425 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:50.705425 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.705425 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:50.705425 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.705425 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:50.706175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:50.706467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:50.706467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:50.706467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.706467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:50.706467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:50.706467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.706467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:50.706467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.706467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:50.706808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.706808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:50.706808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:50.706808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:50.706808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.706808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:50.706808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:50.706808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:00:50.706808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:50.706808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:50.706808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.706808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:50.707044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:50.707292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.707292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:50.707292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.707292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:50.707717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:50.710320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:50.710638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.710638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:50.710638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:50.710638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.710638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:50.710638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.710638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:50.724868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:50.725310 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:50.725400 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:50.725506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:50.726488 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.726711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.727269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:50.727269 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.727269 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:50.727269 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:50.727269 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.727269 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:50.727269 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.727269 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:50.727872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:50.728305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:50.728305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:50.728305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.728305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:50.728305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:50.728305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.728305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:50.728305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.728305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:50.729853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.729853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:50.729853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:50.729853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:50.729853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.729853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:50.729853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:50.729853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:50.729853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:50.729853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.729853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:50.730167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:50.730556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.730556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:50.730556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.730556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.733363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:50.734089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:50.734427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.734427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:50.734427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.734427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:50.737389 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:50.737618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:50.737717 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:50.737861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:50.740134 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.740482 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.743725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:50.744584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:50.744925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:50.745244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:50.745574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:50.745574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:50.745574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:50.745574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:50.745574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:50.745574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:50.745574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:50.745574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:50.745574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:50.745574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:50.745574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:50.745574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:54.164028 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:54.166602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:54.166602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:54.166602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:54.206222 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:54.206222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:54.206424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:54.206424 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:54.206424 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:54.206424 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:54.206424 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:54.206424 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:54.206424 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:54.206424 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:54.207214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:54.207214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:54.207214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:54.207214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:54.207214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:54.207214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:54.207214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:54.207214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:54.207214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:54.207214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:54.207214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:54.207214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:54.207489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:54.207819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:54.207819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:54.209141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:54.209568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:54.209865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:54.209865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:54.209865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:55.651060 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:55.653397 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:55.653397 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:55.654237 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.654466 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.654604 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.654721 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.654836 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.654961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.655066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.655168 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.655288 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.655385 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.655482 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.655585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:55.662140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:55.662608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.663012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:55.663012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:55.663012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.663012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:55.663012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.663012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:55.664174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:55.664644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.665134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:55.665134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.665134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:55.702884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:55.703796 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:55.703796 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:55.703796 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:00:55.704122 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.704351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.705800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:55.705800 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.705800 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:55.705800 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:55.705800 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.705800 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:55.705800 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.705800 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:55.706246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:55.706552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:55.706552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:55.706552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.706552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:55.706552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:55.706552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.706552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:55.706552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.706552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:55.707481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.707481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:55.707481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:55.707481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:55.707481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.707481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:55.707481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:55.707481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:00:55.707481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:55.707481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:55.707481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.707481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:55.707761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:55.708076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.708076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:55.708076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.708076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:55.708974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:55.709321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:55.709692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.709692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:55.709692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:55.709692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.709692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:55.709692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.709692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:55.725369 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:55.726070 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:55.726070 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:55.728553 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:55.728553 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.728553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.729254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:55.729254 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.729254 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:55.729254 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:55.729254 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.729254 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:55.729254 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.729254 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:00:55.730365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:00:55.730701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:55.730701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:00:55.730701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.730701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:00:55.730701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:00:55.730701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.730701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:00:55.730701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.730701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:00:55.732313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.732313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:55.732313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:55.732313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:55.732313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.732313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:55.732313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:55.732313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:55.732313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:55.732313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.732313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:55.732539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:55.732847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:55.733111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:55.733444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:55.733444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:55.733444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:55.733444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:55.733444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.733444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:55.733444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:55.733444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.733444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:55.733444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.733444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:55.737757 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:55.738302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:55.738302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:55.738539 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:55.740042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.740319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:55.740989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:55.743659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:55.744223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.744471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:55.744696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:55.745009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:55.745009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:55.745009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:55.745009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:55.745009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:55.745009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:55.745009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:55.745009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:55.745009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:55.745009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:00:59.164886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:00:59.165498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:00:59.165552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:00:59.166670 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:00:59.236788 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:00:59.236979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:59.237632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:00:59.237632 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:59.237632 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:00:59.237632 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:00:59.237632 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:59.237632 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:00:59.237632 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:59.237632 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:00:59.238413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.072 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:59.238413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:00:59.238413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:59.238413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:00:59.238413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:59.238413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:59.238413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:59.238413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:59.238413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:59.238413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:59.238413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:00:59.238413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:59.238799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:00:59.239158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:59.239158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:00:59.239261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:00:59.239703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:00:59.240261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:00:59.240261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:00:59.240261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:00.652128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:00.652771 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:00.652771 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:00.654125 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.656038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.656038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.656038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.656038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.656038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.656412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.656412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.656412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.656412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.656412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.656412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.656631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:00.656631 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.656631 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:00.656631 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:00.656631 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.656631 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:00.656631 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.656631 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:00.657741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.657741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:00.657741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:00.657741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:00.657741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.657741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:00.657741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:00.657741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.657741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:00.657741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:00.657741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:00.657741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.658152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:00.658517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.658517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:00.660292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:00.662294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.662594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:00.662594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.662594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:00.703375 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:00.703648 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:00.703774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:00.703959 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:01:00.705058 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.705266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.706404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:00.706404 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.706404 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:00.706404 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:00.706404 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.706404 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:00.706404 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.706404 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:00.709389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:00.710275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:00.710275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:00.710275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.710275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:00.710275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:00.710275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.710275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:00.710275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.710275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:00.711088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.711088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:00.711088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:00.711088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:00.711088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.711088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:00.711088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:00.711088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:01:00.711088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:00.711088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:00.711088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.711088 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:00.711836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:00.712416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.712416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:00.712416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.712416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:00.714611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:00.715098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:00.715553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.715553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:00.715553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:00.715553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.715553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:00.715553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.715553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:00.725968 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:00.726314 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:00.726426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:00.726569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:00.727152 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.728019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.728210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:00.728210 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.728210 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:00.728210 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:00.728210 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.728210 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:00.728210 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.728210 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:00.729055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:00.729408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:00.729408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:00.729408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.729408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:00.729408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:00.729408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.729408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:00.729408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.729408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:00.729971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.729971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:00.729971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:00.729971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:00.729971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.729971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:00.729971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:00.729971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:00.729971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:00.729971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.729971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:00.730214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:00.730787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.730787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:00.730787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.730787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.731051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:00.731494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:00.732028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.732028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:00.732028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.732028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:00.738162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:00.742579 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:00.742717 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:00.742854 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:00.744524 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.744524 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:00.744524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.746474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:00.746474 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.746474 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:00.746474 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:00.746474 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.746474 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:00.746474 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.746474 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:00.747552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.747552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:00.747552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:00.747552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:00.747552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.747552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:00.747552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:00.747552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.747552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:00.747552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:00.747552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:00.747552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.748053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:00.748399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.748399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:00.748542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:00.749034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:00.749894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:00.749894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:00.749894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:04.165625 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:04.166242 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:04.166242 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:04.166756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:04.241743 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:04.241982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:04.242557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:04.242557 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:04.242557 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:04.242557 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:04.242557 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:04.242557 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:04.242557 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:04.242557 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:04.243486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.077 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:04.243486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:04.243486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:04.243486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:04.243486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:04.243486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:04.243486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:04.243486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:04.243486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:04.243486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:04.243486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:04.243486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:04.243812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:04.244040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:04.244040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:04.244872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:04.245697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:04.246547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:04.246547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:04.246547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:05.652577 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:05.653239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:05.653304 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:05.656278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.657852 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.657852 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.657852 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.657852 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.658001 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.658102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.661606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.661791 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.661969 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.662101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.662253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.666326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:05.666326 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.666326 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:05.666326 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:05.666326 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.666326 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:05.666326 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.666326 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:05.668073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.013 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.668073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:05.668073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:05.668073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:05.668073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.668073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:05.668073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:05.668073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.668073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:05.668073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:05.668073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:05.668073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.668380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:05.668650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.668650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:05.672124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:05.672549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.672838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:05.672838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.672838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:05.703810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:05.704410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:05.704552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:05.704618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:01:05.705003 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.705235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:05.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:05.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:05.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:05.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.710534 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:05.711521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:05.711824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:05.711824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:05.711824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.711824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:05.711824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:05.711824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.711824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:05.711824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.711824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:05.712833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.712833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:05.712833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:05.712833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:05.712833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.712833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:05.712833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:05.712833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:01:05.712833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:05.712833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:05.712833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.712833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:05.713148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:05.713660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.713660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:05.713660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.713660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:05.714025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:05.714517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:05.715030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.715030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:05.715030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:05.715030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.715030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:05.715030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.715030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:05.726444 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:05.727008 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:05.727119 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:05.727330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:05.727680 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.727888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.728673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:05.728673 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.728673 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:05.728673 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:05.728673 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.728673 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:05.728673 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.728673 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:05.729521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:05.729912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:05.729912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:05.729912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.729912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:05.729912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:05.729912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.729912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:05.729912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.729912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:05.730500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.730500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:05.730500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:05.730500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:05.730500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.730500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:05.730500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:05.730500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:05.730500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:05.730500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.730500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:05.730811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:05.731156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.731156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:05.731156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.731156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.731627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:05.732322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:05.732777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.732777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:05.732777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.732777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:05.742660 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:05.742950 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:05.743088 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:05.743289 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:05.743881 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.744158 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:05.744509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.745373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:05.745373 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.745373 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:05.745373 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:05.745373 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.745373 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:05.745373 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.745373 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:05.746314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.746314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:05.746314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:05.746314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:05.746314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.746314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:05.746314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:05.746314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.746314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:05.746314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:05.746314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:05.746314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.746707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:05.747075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.747075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:05.747441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:05.747853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:05.748177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:05.748177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:05.748177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:09.165849 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:09.166512 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:09.166512 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:09.166915 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:09.199982 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:09.200232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:09.200642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:09.200642 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:09.200642 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:09.200642 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:09.200642 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:09.200642 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:09.200642 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:09.200642 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:09.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:09.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:09.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:09.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:09.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:09.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:09.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:09.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:09.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:09.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:09.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:09.201389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:09.201753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:09.202128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:09.202561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:09.203061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:09.203061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:09.203061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:09.203061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:09.203061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:09.203061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:10.653099 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:10.654069 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:10.656868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:10.657580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.657752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.657909 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.658020 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.658126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.658243 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.658354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.658475 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.658593 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.658692 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.658796 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.658950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.659425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:10.659425 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.659425 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:10.659425 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:10.659425 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.659425 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:10.659425 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.659425 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:10.660241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.660241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:10.660241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:10.660241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:10.660241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.660241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:10.660241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:10.660241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.660241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:10.660241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:10.660241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:10.660241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.660646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:10.660988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.660988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:10.661223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:10.661612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.662105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:10.662105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.662105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:10.707376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:10.707613 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:10.707718 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:10.708868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:01:10.708868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.709434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.709489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:10.709489 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.709489 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:10.709489 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:10.709489 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.709489 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:10.709489 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.709489 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:10.710299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:10.710594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:10.710594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:10.710594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.710594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:10.710594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:10.710594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.710594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:10.710594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.710594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:10.713162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.713162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:10.713162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:10.713162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:10.713162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.713162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:10.713162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:10.713162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:01:10.713162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:10.713162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:10.713162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.713162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:10.713503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:10.713779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:10.714017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:10.714377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:10.714377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:10.714377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:10.714377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:10.714377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:10.714377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.714377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:10.714377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:10.714377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.714377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:10.714377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.714377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:10.728508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:10.729506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:10.729696 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:10.729696 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:10.733281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.733281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.733497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:10.733497 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.733497 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:10.733497 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:10.733497 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.733497 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:10.733497 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.733497 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:10.734733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:10.735136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:10.735136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:10.735136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.735136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:10.735136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:10.735136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.735136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:10.735136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.735136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:10.738216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.738216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:10.738216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:10.738216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:10.738216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.738216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:10.738216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:10.738216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:10.738216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:10.738216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.738216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:10.738745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:10.739131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.739131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:10.739131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.739131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.739600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:10.740082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:10.740570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.740570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:10.740570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.740570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:10.743236 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:10.744071 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:10.744071 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:10.744235 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:10.744509 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.746316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:10.746316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.746457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:10.746457 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.746457 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:10.746457 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:10.746457 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.746457 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:10.746457 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.746457 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:10.748361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.748361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:10.748361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:10.748361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:10.748361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.748361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:10.748361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:10.748361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.748361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:10.748361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:10.748361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:10.748361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.748607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:10.748794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.748794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:10.749471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:10.750463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:10.750838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:10.750838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:10.750838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:14.166320 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:14.166738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:14.166738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:14.167269 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:14.213407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:14.214241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:14.214702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:14.214702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:14.214702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:14.214702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:14.214702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:14.214702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:14.214702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:14.214702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:14.214702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:14.214702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:14.214702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:14.214702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:14.215078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:14.215640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:15.655467 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:15.656348 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:15.656500 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:15.657625 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.657840 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-415af2dc-87b5-455f-b1a4-8ddde2eb7ddc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.657983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.658114 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.658479 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.658479 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.658479 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.658642 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.658739 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.658927 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.659110 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-681f9149-16aa-4b8f-beb8-6b92d66e15c5, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.659223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.660909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:15.660909 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.660909 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:15.660909 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:15.660909 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.660909 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:15.660909 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.660909 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:15.663478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.663478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:15.663478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:15.663478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:15.663478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.663478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:15.663478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:15.663478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.663478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:15.663478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:15.663478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:15.663478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.663850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:15.664543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:15.664820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.665164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:15.665164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:15.665164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.665164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:15.665164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.665164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:15.707752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:15.708094 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:15.708230 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:15.708446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:01:15.709341 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.709510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.710209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:15.710209 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.710209 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:15.710209 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:15.710209 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.710209 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:15.710209 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.710209 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:15.711062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:15.711381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:15.711381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:15.711381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.711381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:15.711381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:15.711381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.711381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:15.711381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.711381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:15.712046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.712046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:15.712046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:15.712046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:15.712046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.712046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:15.712046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:15.712046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:01:15.712046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:15.712046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:15.712046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.712046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:15.712334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:15.712593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.712593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:15.712593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.712593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:15.713157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:15.713557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:15.713978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.713978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:15.713978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:15.713978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.713978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:15.713978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.713978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:15.729636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:15.730326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:15.730443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:15.730597 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:15.731290 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.731493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.732406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:15.732406 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.732406 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:15.732406 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:15.732406 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.732406 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:15.732406 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.732406 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:15.734424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:15.734696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:15.734696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:15.734696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.734696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:15.734696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:15.734696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.734696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:15.734696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.734696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:15.737371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.737371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:15.737371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:15.737371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:15.737371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.737371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:15.737371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:15.737371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:15.737371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:15.737371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.737371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:15.737740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:15.737963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.737963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:15.737963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.737963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.739329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:15.739675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:15.739993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.739993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:15.739993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.739993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:15.744101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:15.749677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:15.749746 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:15.749915 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:15.752466 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.752869 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:15.752978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.753787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:15.753787 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.753787 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:15.753787 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:15.753787 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.753787 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:15.753787 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.753787 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:15.759020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.759020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:15.759020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:15.759020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:15.759020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.759020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:15.759020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:15.759020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.759020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:15.759020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:15.759020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:15.759020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.759765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:15.760032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.760032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:15.762529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:15.762850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:15.763153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:15.763153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:15.763153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:19.166768 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:19.167496 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:19.167496 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:19.167683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:19.187315 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=247 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:01:19.206907 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:19.207181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:19.207980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:19.207980 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:19.207980 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:19.207980 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:19.207980 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:19.207980 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:19.207980 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:19.207980 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:19.208769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:19.208769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:19.208769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:19.208769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:19.208769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:19.208769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:19.208769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:19.208769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:19.208769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:19.208769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:19.208769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:19.208769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:19.209151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:19.209472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:19.209869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:19.210395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:19.210395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:19.210395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:19.210395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:19.210395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:19.210395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:20.657488 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:20.658087 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:20.658087 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:20.659756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.659756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.659909 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.660041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.660165 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.660394 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.660517 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.660681 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.661595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.661595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.661687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:20.661687 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.661687 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:20.661687 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:20.661687 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.661687 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:20.661687 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.661687 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:20.662638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.662638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:20.662638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:20.662638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:20.662638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.662638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:20.662638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:20.662638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.662638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:20.662638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:20.662638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:20.662638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.663036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:20.663454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.663454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:20.664860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:20.665458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.666944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:20.666944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.666944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:20.708214 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:20.708522 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:20.708656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:20.708778 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:01:20.709261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.709564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.710144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:20.710144 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.710144 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:20.710144 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:20.710144 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.710144 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:20.710144 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.710144 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:20.710966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:20.711361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:20.711361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:20.711361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.711361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:20.711361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:20.711361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.711361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:20.711361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.711361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:20.711699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.711699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:20.711699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:20.711699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:20.711699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.711699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:20.711699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:20.711699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:01:20.711699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:20.711699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:20.711699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.711699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:20.711908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:20.712265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.712265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:20.712265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.712265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:20.712436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:20.712824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:20.713295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.713295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:20.713295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:20.713295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.713295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:20.713295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.713295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:20.731734 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:20.731734 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:20.731734 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:20.732150 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:20.733701 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.734019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.735398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.736052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:20.739019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.739019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:20.739019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:20.739019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:20.739019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.739019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:20.739019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:20.739019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:20.739019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:20.739019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.739019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:20.739391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:20.739765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.739765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:20.739765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.739765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.741230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:20.741566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:20.741878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.741878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:20.741878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.741878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:20.749996 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:20.750382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:20.751422 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:20.751677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:20.753098 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.753098 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:20.753098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.753565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:20.753565 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.753565 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:20.753565 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:20.753565 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.753565 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:20.753565 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.753565 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:20.755291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.755291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:20.755291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:20.755291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:20.755291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.755291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:20.755291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:20.755291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.755291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:20.755291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:20.755291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:20.755291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.755613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:20.755978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:20.756288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:20.757084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:20.757084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:20.757084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:20.757084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:20.757084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:20.757084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:24.158325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:24.158803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:24.158803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:24.168145 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:24.170120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 09 01:01:24.170333 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Number of inconsistencies found at create/update: networks=9, security_groups=3, routers=6, ports=1, router_ports=6 {{(pid=89255) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 09 01:01:24.170477 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=20 {{(pid=89255) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 09 01:01:24.170578 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource b3e28628-ad68-4ca3-b181-b39c995a69e0 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.208131 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, resource={'id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'name': 'public', 'tenant_id': '5a148cfd27844ee6ad0d79cb077e4985', 'project_id': '5a148cfd27844ee6ad0d79cb077e4985', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['17135a44-456c-43be-8f7d-7714aa3731cf', 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-09T00:51:13Z', 'updated_at': '2026-06-09T00:51:20Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.208419 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.208633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-6a91ff68-89ba-411b-b54f-ff58910954a3, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.209150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.209262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:24.209262 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.209262 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:24.209262 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:24.209262 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.209262 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:24.209262 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.209262 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:24.210729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource b3e28628-ad68-4ca3-b181-b39c995a69e0 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.210729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:24.210729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:24.210729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:24.210729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:24.210729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:24.210729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:01:24.210729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:24.210729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.210729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.210729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:24.211122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.211694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:24.211694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.211694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:24.211694 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 99ed7e2d-1377-4fcf-97e4-276b166edce7 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.239013 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-99ed7e2d-1377-4fcf-97e4-276b166edce7, resource={'id': '99ed7e2d-1377-4fcf-97e4-276b166edce7', 'name': 'tempest-AttachVolumeTestJSON-1161663411-network', 'tenant_id': '6aa0a579fc4046c182c42afe260ffe6b', 'project_id': '6aa0a579fc4046c182c42afe260ffe6b', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['8c1faa4d-2807-40c6-a617-73948b7871a1'], 'standard_attr_id': 882, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:00:09Z', 'updated_at': '2026-06-09T01:00:09Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 53995}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.239332 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-99ed7e2d-1377-4fcf-97e4-276b166edce7, col_values=(('external_ids', {'neutron:network_name': 'tempest-AttachVolumeTestJSON-1161663411-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.239690 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-ba39cb4b-f6e8-437a-aa9b-c4f67353e030, external_ids_update=None, columns={'tag': 53995}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.239863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.240462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:24.240462 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.240462 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:24.240462 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:24.240462 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.240462 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:24.240462 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.240462 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:24.241682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 99ed7e2d-1377-4fcf-97e4-276b166edce7 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.241682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:24.241682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:24.241682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:24.241682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:24.241682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:24.241682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:01:24.241682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:24.241682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.241682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.241682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:24.242159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.242433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:24.242433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.242433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:24.242433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 404ab349-196e-441d-9b05-a49587ddbd2d (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.277109 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-404ab349-196e-441d-9b05-a49587ddbd2d, resource={'id': '404ab349-196e-441d-9b05-a49587ddbd2d', 'name': 'tempest-TestShelveInstance-566557053-network', 'tenant_id': 'aa4e043d2ccd419abf9f9d2f79c69b9d', 'project_id': 'aa4e043d2ccd419abf9f9d2f79c69b9d', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['faf5fa4c-f1d4-4533-9629-cc4aaaac6d62'], 'standard_attr_id': 839, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:59:59Z', 'updated_at': '2026-06-09T00:59:59Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 9912}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.277499 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-404ab349-196e-441d-9b05-a49587ddbd2d, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestShelveInstance-566557053-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.277953 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-64460592-5ce6-4b3c-a8fa-10b2091accfb, external_ids_update=None, columns={'tag': 9912}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.278067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.278706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:24.278706 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.278706 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:24.278706 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:24.278706 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.278706 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:24.278706 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.278706 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:24.280022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 404ab349-196e-441d-9b05-a49587ddbd2d (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.280022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:24.280022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:24.280022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:24.280022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:24.280022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:24.280022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:01:24.280022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:24.280022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.280022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.280022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:24.280390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.280666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:24.280666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.280666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:24.280666 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 214cff6f-a123-4b5a-bb21-0888baae6644 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.315992 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-214cff6f-a123-4b5a-bb21-0888baae6644, resource={'id': '214cff6f-a123-4b5a-bb21-0888baae6644', 'name': 'tempest-TestShelveInstance-695374279-network', 'tenant_id': '1d25beb469d54997b4ba8cd39e6c344b', 'project_id': '1d25beb469d54997b4ba8cd39e6c344b', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['d07ba610-d7e7-480b-92d1-3dcb7ba42bf7'], 'standard_attr_id': 807, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:59:52Z', 'updated_at': '2026-06-09T00:59:53Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 46074}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.316264 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-214cff6f-a123-4b5a-bb21-0888baae6644, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestShelveInstance-695374279-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.316491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-b29f9b5e-fad0-4d7f-bd15-4d72470b01e5, external_ids_update=None, columns={'tag': 46074}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.316602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.317395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:24.317395 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.317395 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:24.317395 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:24.317395 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.317395 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:24.317395 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.317395 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:24.318691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 214cff6f-a123-4b5a-bb21-0888baae6644 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.318691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:24.318691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:24.318691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:24.318691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:24.318691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:24.318691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:01:24.318691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:24.318691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.318691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.318691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:24.318911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.319244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:24.319244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.319244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:24.319244 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource b6af5245-0d30-4174-8d5f-6eaf6030b844 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.354422 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-b6af5245-0d30-4174-8d5f-6eaf6030b844, resource={'id': 'b6af5245-0d30-4174-8d5f-6eaf6030b844', 'name': 'tempest-ServerActionsTestOtherB-248196719-network', 'tenant_id': 'b3807c1b7608434fb7dd582c26bcbc84', 'project_id': 'b3807c1b7608434fb7dd582c26bcbc84', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['f36550c4-3e1f-4ef0-8306-99b05777cac6'], 'standard_attr_id': 790, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:59:49Z', 'updated_at': '2026-06-09T00:59:50Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 45305}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.354693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-b6af5245-0d30-4174-8d5f-6eaf6030b844, col_values=(('external_ids', {'neutron:network_name': 'tempest-ServerActionsTestOtherB-248196719-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.354921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-c1ac3ed1-e4f1-4d66-9ef8-7bbbba592879, external_ids_update=None, columns={'tag': 45305}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.355024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.355559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:24.355559 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.355559 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:24.355559 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:24.355559 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.355559 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:24.355559 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.355559 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:24.356479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource b6af5245-0d30-4174-8d5f-6eaf6030b844 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.356479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:24.356479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:24.356479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:24.356479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:24.356479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:24.356479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:01:24.356479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:24.356479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.356479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.356479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:24.356829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.357225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:24.357225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.357225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:24.357225 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 936c7a5d-b3be-4ccb-88ed-abf688e4b7ae (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.384944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-936c7a5d-b3be-4ccb-88ed-abf688e4b7ae, resource={'id': '936c7a5d-b3be-4ccb-88ed-abf688e4b7ae', 'name': 'tempest-ServerRescueNegativeTestJSON-973928879-network', 'tenant_id': 'a8deb93277d848b78534fb0b0e7ecd28', 'project_id': 'a8deb93277d848b78534fb0b0e7ecd28', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['de82e60b-3ff3-4e59-8859-fb0054bcf9ff'], 'standard_attr_id': 771, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:59:41Z', 'updated_at': '2026-06-09T00:59:42Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 26686}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.385079 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-936c7a5d-b3be-4ccb-88ed-abf688e4b7ae, col_values=(('external_ids', {'neutron:network_name': 'tempest-ServerRescueNegativeTestJSON-973928879-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.385408 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-7348fa50-cf2f-433c-862a-91e0dc3092d5, external_ids_update=None, columns={'tag': 26686}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.385495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.386026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:24.386026 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.386026 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:24.386026 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:24.386026 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.386026 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:24.386026 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.386026 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:24.386815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 936c7a5d-b3be-4ccb-88ed-abf688e4b7ae (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.386815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:24.386815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:24.386815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:24.386815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:24.386815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:24.386815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:01:24.386815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:24.386815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.386815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.386815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:24.387051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.387307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:24.387307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.387307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:24.387307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource dc15bda6-1b51-4a9e-af84-ac570d199c11 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.418662 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-dc15bda6-1b51-4a9e-af84-ac570d199c11, resource={'id': 'dc15bda6-1b51-4a9e-af84-ac570d199c11', 'name': 'private', 'tenant_id': '5a148cfd27844ee6ad0d79cb077e4985', 'project_id': '5a148cfd27844ee6ad0d79cb077e4985', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['65e82083-c074-4bc8-b13e-2eeabecc299d', '6f142de2-8acf-46b2-af54-c34cf691eff5'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:51:08Z', 'updated_at': '2026-06-09T00:51:11Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 62028}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.418903 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-dc15bda6-1b51-4a9e-af84-ac570d199c11, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.419168 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-77925902-9e1b-4f7c-868e-70a5d73ee785, external_ids_update=None, columns={'tag': 62028}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.419286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.419831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:24.419831 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.419831 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:24.419831 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:24.419831 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.419831 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:24.419831 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.419831 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:24.420701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource dc15bda6-1b51-4a9e-af84-ac570d199c11 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.420701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:24.420701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:24.420701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:24.420701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:24.420701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:24.420701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:01:24.420701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:24.420701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.420701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.420701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:24.420955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.421182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:24.421182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.421182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:24.421182 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 84d53e7b-f1f4-4dcd-a163-e19ff885797e (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.452425 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-84d53e7b-f1f4-4dcd-a163-e19ff885797e, resource={'id': '84d53e7b-f1f4-4dcd-a163-e19ff885797e', 'name': 'tempest-AttachVolumeNegativeTest-1148979953-network', 'tenant_id': '0d1935a7ccae45788150f267d0103d61', 'project_id': '0d1935a7ccae45788150f267d0103d61', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['4dc164e2-fb83-414d-8af5-52f40f62911c'], 'standard_attr_id': 606, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:57:52Z', 'updated_at': '2026-06-09T00:57:53Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 43381}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.452741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-84d53e7b-f1f4-4dcd-a163-e19ff885797e, col_values=(('external_ids', {'neutron:network_name': 'tempest-AttachVolumeNegativeTest-1148979953-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.453059 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-bbd844bf-723d-4230-b13f-79db3737d3bf, external_ids_update=None, columns={'tag': 43381}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.453203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.453893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:24.453893 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.453893 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:24.453893 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:24.453893 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.453893 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:24.453893 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.453893 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:24.454861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 84d53e7b-f1f4-4dcd-a163-e19ff885797e (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.454861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:24.454861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:24.454861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:24.454861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:24.454861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:24.454861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:01:24.454861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:24.454861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.454861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.454861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:24.455233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.455698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:24.455698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.455698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:24.455698 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 05419c49-9db5-4d5d-8bbe-5a7016c12d7e (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.491020 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-05419c49-9db5-4d5d-8bbe-5a7016c12d7e, resource={'id': '05419c49-9db5-4d5d-8bbe-5a7016c12d7e', 'name': 'shared', 'tenant_id': '5a148cfd27844ee6ad0d79cb077e4985', 'project_id': '5a148cfd27844ee6ad0d79cb077e4985', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['612c5f15-747e-4ea8-a2d1-d0bbb8f1bc11'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:52:24Z', 'updated_at': '2026-06-09T00:52:25Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 44634}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.491476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-05419c49-9db5-4d5d-8bbe-5a7016c12d7e, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.491556 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-1c288345-932b-445f-8f63-edb47405877c, external_ids_update=None, columns={'tag': 44634}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.491644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.492173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:24.492173 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.492173 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:24.492173 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:24.492173 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.492173 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:24.492173 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.492173 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:24.493028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 05419c49-9db5-4d5d-8bbe-5a7016c12d7e (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.493028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:24.493028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:24.493028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:24.493028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:24.493028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:24.493028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:01:24.493028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:24.493028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.493028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.493028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:24.493347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.493693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:24.493693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.493693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:24.493693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource fc110cda-f93f-444e-ba08-630fcbe98de5 (type: security_groups) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.517318 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Successfully bumped revision number for resource fc110cda-f93f-444e-ba08-630fcbe98de5 (type: security_groups) to 3 Jun 09 01:01:24.517318 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 76fd1df8-0bfb-43c7-9cfd-99b3e998b904 (type: security_groups) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.546019 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Successfully bumped revision number for resource 76fd1df8-0bfb-43c7-9cfd-99b3e998b904 (type: security_groups) to 7 Jun 09 01:01:24.546238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 44108f5f-8d38-450c-8768-0190dfce6cf2 (type: security_groups) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.568746 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Successfully bumped revision number for resource 44108f5f-8d38-450c-8768-0190dfce6cf2 (type: security_groups) to 3 Jun 09 01:01:24.568907 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource d463cd7c-6ca6-4c0e-825e-4d41c90a60f9 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.656393 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, resource={'id': 'd463cd7c-6ca6-4c0e-825e-4d41c90a60f9', 'name': 'tempest-ServerRescueNegativeTestJSON-973928879-router', 'project_id': 'a8deb93277d848b78534fb0b0e7ecd28', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.17'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::348'}], 'enable_snat': True}, 'gw_port_id': '175878c3-ae5a-4210-859f-45696c6b4999', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.17', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::348', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T00:59:42Z', 'updated_at': '2026-06-09T00:59:45Z', 'revision_number': 3, 'tenant_id': 'a8deb93277d848b78534fb0b0e7ecd28'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.657632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, columns={'external_ids': {'neutron:router_name': 'tempest-ServerRescueNegativeTestJSON-973928879-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.657632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "175878c3-ae5a-4210-859f-45696c6b4999"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.657632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "175878c3-ae5a-4210-859f-45696c6b4999"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.657632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.657632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:24.657632 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.657632 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:24.657632 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:24.658205 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.658205 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:24.658205 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.658205 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:24.658205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router d463cd7c-6ca6-4c0e-825e-4d41c90a60f9. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.661697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource d463cd7c-6ca6-4c0e-825e-4d41c90a60f9 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.661697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:24.661697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:24.661697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:24.661697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:24.661697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:24.661697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:01:24.661697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:01:24.661697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:01:24.661697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:01:24.661697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:24.662104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:24.662514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:24.662514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:24.662514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.662514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:24.662514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:24.662514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.662514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:24.662514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.662514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:24.662514 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 65f33571-0494-499f-be92-d5ac93b22e88 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.755123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-65f33571-0494-499f-be92-d5ac93b22e88, resource={'id': '65f33571-0494-499f-be92-d5ac93b22e88', 'name': 'tempest-AttachVolumeTestJSON-1161663411-router', 'project_id': '6aa0a579fc4046c182c42afe260ffe6b', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.139'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::d8'}], 'enable_snat': True}, 'gw_port_id': 'f612d464-2a07-4d0d-8d1b-99a5d66c807b', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.139', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::d8', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:00:10Z', 'updated_at': '2026-06-09T01:00:13Z', 'revision_number': 3, 'tenant_id': '6aa0a579fc4046c182c42afe260ffe6b'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.755340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-65f33571-0494-499f-be92-d5ac93b22e88, columns={'external_ids': {'neutron:router_name': 'tempest-AttachVolumeTestJSON-1161663411-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.755398 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "f612d464-2a07-4d0d-8d1b-99a5d66c807b"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.756248 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "f612d464-2a07-4d0d-8d1b-99a5d66c807b"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.756248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.757132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:24.757132 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.757132 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:24.757132 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:24.757132 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.757132 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:24.757132 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.757132 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:24.757392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 65f33571-0494-499f-be92-d5ac93b22e88. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.758732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 65f33571-0494-499f-be92-d5ac93b22e88 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.758732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:24.758732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:24.758732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:24.758732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:24.758732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:24.758732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:01:24.758732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:01:24.758732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:01:24.758732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:01:24.758732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:24.759399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:24.760302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:24.760302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:24.760302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.760302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:24.760302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:24.760302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.760302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:24.760302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.760302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:24.760302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 31f8b7b1-49f4-4311-958d-2e3c14344865 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:24.865781 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, resource={'id': '31f8b7b1-49f4-4311-958d-2e3c14344865', 'name': 'tempest-TestShelveInstance-566557053-router', 'project_id': 'aa4e043d2ccd419abf9f9d2f79c69b9d', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.61'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::104'}], 'enable_snat': True}, 'gw_port_id': 'dad22407-98da-463e-b319-a4d46802b839', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.61', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::104', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:00:00Z', 'updated_at': '2026-06-09T01:00:04Z', 'revision_number': 3, 'tenant_id': 'aa4e043d2ccd419abf9f9d2f79c69b9d'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.866815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, columns={'external_ids': {'neutron:router_name': 'tempest-TestShelveInstance-566557053-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.869090 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "dad22407-98da-463e-b319-a4d46802b839"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.869525 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "dad22407-98da-463e-b319-a4d46802b839"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:24.871596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.873267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:24.873267 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.873267 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:24.873267 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:24.873267 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.873267 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:24.873267 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.873267 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:24.873267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 31f8b7b1-49f4-4311-958d-2e3c14344865. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.877598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 31f8b7b1-49f4-4311-958d-2e3c14344865 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.877598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:24.877598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:24.877598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:24.877598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:24.877598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:24.877598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:01:24.877598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:01:24.877598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:01:24.877598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:01:24.877598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:24.878030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:24.878362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:24.878362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:24.878362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:24.878362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:24.878362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:24.878362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:24.878362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:24.878362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:24.878362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:24.878559 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 8910a7d6-bd8b-4f14-9218-1eb03eb60a13 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:25.002276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, resource={'id': '8910a7d6-bd8b-4f14-9218-1eb03eb60a13', 'name': 'tempest-TestShelveInstance-695374279-router', 'project_id': '1d25beb469d54997b4ba8cd39e6c344b', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.203'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::98'}], 'enable_snat': True}, 'gw_port_id': '95bf70ca-58e3-4fae-a612-fe561367e0cc', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.203', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::98', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T00:59:55Z', 'updated_at': '2026-06-09T00:59:57Z', 'revision_number': 3, 'tenant_id': '1d25beb469d54997b4ba8cd39e6c344b'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.002499 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, columns={'external_ids': {'neutron:router_name': 'tempest-TestShelveInstance-695374279-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.002586 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "95bf70ca-58e3-4fae-a612-fe561367e0cc"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.002706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "95bf70ca-58e3-4fae-a612-fe561367e0cc"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.002768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.003422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:25.003422 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.003422 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:25.003422 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:25.003422 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.003422 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:25.003422 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.003422 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:25.003918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 8910a7d6-bd8b-4f14-9218-1eb03eb60a13. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.005371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 8910a7d6-bd8b-4f14-9218-1eb03eb60a13 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.005371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:25.005371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:25.005371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:25.005371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:25.005371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:25.005371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:01:25.005371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:01:25.005371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:01:25.005371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:01:25.005371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:25.005654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:25.005897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:25.005897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:25.005897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.005897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:25.005897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:25.005897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.005897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:25.005897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.005897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:25.005897 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 639a20b9-5f39-4f38-8396-d47ebfd69416 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:25.100579 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, resource={'id': '639a20b9-5f39-4f38-8396-d47ebfd69416', 'name': 'tempest-ServerActionsTestOtherB-248196719-router', 'project_id': 'b3807c1b7608434fb7dd582c26bcbc84', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.195'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::5c'}], 'enable_snat': True}, 'gw_port_id': 'ec1323ed-b9b5-4b00-a74e-fe7ee50211ed', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.195', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::5c', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T00:59:51Z', 'updated_at': '2026-06-09T00:59:53Z', 'revision_number': 3, 'tenant_id': 'b3807c1b7608434fb7dd582c26bcbc84'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.100823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, columns={'external_ids': {'neutron:router_name': 'tempest-ServerActionsTestOtherB-248196719-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.101035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "ec1323ed-b9b5-4b00-a74e-fe7ee50211ed"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.101202 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "ec1323ed-b9b5-4b00-a74e-fe7ee50211ed"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.101329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.102008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:25.102008 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.102008 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:25.102008 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:25.102008 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.102008 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:25.102008 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.102008 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:25.102244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 639a20b9-5f39-4f38-8396-d47ebfd69416. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.103810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 639a20b9-5f39-4f38-8396-d47ebfd69416 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.103810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:25.103810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:25.103810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:25.103810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:25.103810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:25.103810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:01:25.103810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:01:25.103810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:01:25.103810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:01:25.103810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:25.104021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:25.104327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:25.104327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:25.104327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.104327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:25.104327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:25.104327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.104327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:25.104327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.104327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:25.104327 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 7abe82fe-be95-418c-ba05-6ac3294f95e7 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:25.184652 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, resource={'id': '7abe82fe-be95-418c-ba05-6ac3294f95e7', 'name': 'tempest-AttachVolumeNegativeTest-1148979953-router', 'project_id': '0d1935a7ccae45788150f267d0103d61', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.38'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::264'}], 'enable_snat': True}, 'gw_port_id': '4a8b370e-f68f-43ed-939a-0f29e8a361f2', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.38', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::264', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T00:57:54Z', 'updated_at': '2026-06-09T00:57:57Z', 'revision_number': 3, 'tenant_id': '0d1935a7ccae45788150f267d0103d61'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.185251 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, columns={'external_ids': {'neutron:router_name': 'tempest-AttachVolumeNegativeTest-1148979953-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.185251 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "4a8b370e-f68f-43ed-939a-0f29e8a361f2"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.185251 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "4a8b370e-f68f-43ed-939a-0f29e8a361f2"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.185251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.185251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:25.185251 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.185251 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:25.185251 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:25.185489 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.185489 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:25.185489 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.185489 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:25.185489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 7abe82fe-be95-418c-ba05-6ac3294f95e7. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.187051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 7abe82fe-be95-418c-ba05-6ac3294f95e7 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.187051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:25.187051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:25.187051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:25.187051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:25.187051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:25.187051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:01:25.187051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:01:25.187051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:01:25.187051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:01:25.187051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:25.187579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:25.187819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:25.187819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:25.187819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.187819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:25.187819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:25.187819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.187819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:25.187819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.187819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:25.187819 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource e16f7f2b-e9bd-41c2-bcba-557457e1246e (type: ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:25.271007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=e16f7f2b-e9bd-41c2-bcba-557457e1246e, resource={'id': 'e16f7f2b-e9bd-41c2-bcba-557457e1246e', 'name': '', 'network_id': '214cff6f-a123-4b5a-bb21-0888baae6644', 'project_id': '1d25beb469d54997b4ba8cd39e6c344b', 'mac_address': 'fa:16:3e:40:87:87', 'admin_state_up': True, 'status': 'DOWN', 'device_id': '4dfa9cdb-f39b-411b-a248-7d51b08cfbff', 'device_owner': 'compute:nova', 'standard_attr_id': 901, 'fixed_ips': [{'subnet_id': 'd07ba610-d7e7-480b-92d1-3dcb7ba42bf7', 'ip_address': '10.1.0.6'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['76fd1df8-0bfb-43c7-9cfd-99b3e998b904'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': '', 'binding:vif_type': 'unbound', 'binding:vif_details': {}, 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:00:17Z', 'updated_at': '2026-06-09T01:01:14Z', 'revision_number': 9, 'tenant_id': '1d25beb469d54997b4ba8cd39e6c344b'}, resource_type=ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.271304 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLSwitchPortCommand(_result=None, lport=e16f7f2b-e9bd-41c2-bcba-557457e1246e, external_ids_update=None, columns={'external_ids': {'neutron:port_name': '', 'neutron:device_id': '4dfa9cdb-f39b-411b-a248-7d51b08cfbff', 'neutron:project_id': '1d25beb469d54997b4ba8cd39e6c344b', 'neutron:cidrs': '10.1.0.6/28', 'neutron:device_owner': 'compute:nova', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': '', 'neutron:network_name': 'neutron-214cff6f-a123-4b5a-bb21-0888baae6644', 'neutron:security_group_ids': '76fd1df8-0bfb-43c7-9cfd-99b3e998b904', 'neutron:revision_number': '9', 'neutron:vnic_type': 'normal', 'neutron:port_capabilities': '', 'neutron:mtu': '', 'neutron:port_fip': '172.24.5.64'}, 'parent_name': [], 'tag': [], 'options': {}, 'enabled': True, 'port_security': ['fa:16:3e:40:87:87 10.1.0.6'], 'dhcpv4_options': [UUID('d8828f31-8690-46c0-99a6-4542318c52ff')], 'dhcpv6_options': [], 'type': '', 'addresses': ['fa:16:3e:40:87:87 10.1.0.6'], 'ha_chassis_group': []}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.271774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): PgAddPortCommand(_result=None, port_group=neutron_pg_drop, lsp=['e16f7f2b-e9bd-41c2-bcba-557457e1246e'], if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.271990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.272539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:25.272539 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.272539 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:25.272539 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:25.272539 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.272539 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:25.272539 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.272539 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:25.273824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource e16f7f2b-e9bd-41c2-bcba-557457e1246e (type: ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.273824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:25.273824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:25.273824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:25.273824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:25.273824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:25.273824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 713, in update_port Jun 09 01:01:25.273824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True, Jun 09 01:01:25.273824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.273824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.273824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:25.274212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.274621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:25.274621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.274621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:25.274621 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 95bf70ca-58e3-4fae-a612-fe561367e0cc (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:25.378732 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-95bf70ca-58e3-4fae-a612-fe561367e0cc, resource={'id': '95bf70ca-58e3-4fae-a612-fe561367e0cc', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:09:f2:46', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '8910a7d6-bd8b-4f14-9218-1eb03eb60a13', 'device_owner': 'network:router_gateway', 'standard_attr_id': 813, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.203'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::98'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T00:59:55Z', 'updated_at': '2026-06-09T01:00:01Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.379047 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-95bf70ca-58e3-4fae-a612-fe561367e0cc, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::98/64', '172.24.5.203/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.379318 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=95bf70ca-58e3-4fae-a612-fe561367e0cc, lrouter_port=lrp-95bf70ca-58e3-4fae-a612-fe561367e0cc, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.379488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.380251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:25.380251 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.380251 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:25.380251 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:25.380251 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.380251 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:25.380251 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.380251 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:25.381303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 95bf70ca-58e3-4fae-a612-fe561367e0cc (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.381303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:25.381303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:25.381303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:25.381303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:25.381303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:25.381303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:01:25.381303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:25.381303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.381303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.381303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:25.381616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.381939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:25.381939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.381939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:25.381939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource ec1323ed-b9b5-4b00-a74e-fe7ee50211ed (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:25.484173 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-ec1323ed-b9b5-4b00-a74e-fe7ee50211ed, resource={'id': 'ec1323ed-b9b5-4b00-a74e-fe7ee50211ed', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:60:ea:96', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '639a20b9-5f39-4f38-8396-d47ebfd69416', 'device_owner': 'network:router_gateway', 'standard_attr_id': 800, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.195'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::5c'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T00:59:51Z', 'updated_at': '2026-06-09T01:00:03Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.484460 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-ec1323ed-b9b5-4b00-a74e-fe7ee50211ed, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-639a20b9-5f39-4f38-8396-d47ebfd69416'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.195/24', '2001:db8::5c/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.484709 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=ec1323ed-b9b5-4b00-a74e-fe7ee50211ed, lrouter_port=lrp-ec1323ed-b9b5-4b00-a74e-fe7ee50211ed, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.484885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.485486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:25.485486 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.485486 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:25.485486 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:25.485486 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.485486 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:25.485486 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.485486 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:25.486666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource ec1323ed-b9b5-4b00-a74e-fe7ee50211ed (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.486666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:25.486666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:25.486666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:25.486666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:25.486666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:25.486666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:01:25.486666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:25.486666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.486666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.486666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:25.487074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.487386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:25.487386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.487386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:25.487386 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource dad22407-98da-463e-b319-a4d46802b839 (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:25.572128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-dad22407-98da-463e-b319-a4d46802b839, resource={'id': 'dad22407-98da-463e-b319-a4d46802b839', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:6a:b0:25', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '31f8b7b1-49f4-4311-958d-2e3c14344865', 'device_owner': 'network:router_gateway', 'standard_attr_id': 854, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.61'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::104'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:00:01Z', 'updated_at': '2026-06-09T01:00:04Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.572460 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-dad22407-98da-463e-b319-a4d46802b839, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-31f8b7b1-49f4-4311-958d-2e3c14344865'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::104/64', '172.24.5.61/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.572704 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=dad22407-98da-463e-b319-a4d46802b839, lrouter_port=lrp-dad22407-98da-463e-b319-a4d46802b839, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.572906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.573501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:25.573501 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.573501 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:25.573501 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:25.573501 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.573501 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:25.573501 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.573501 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:25.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource dad22407-98da-463e-b319-a4d46802b839 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:25.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:25.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:25.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:25.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:25.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:01:25.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:25.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.574400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:25.574614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.574849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:25.574849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.574849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:25.574849 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 175878c3-ae5a-4210-859f-45696c6b4999 (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:25.659140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:25.663805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-175878c3-ae5a-4210-859f-45696c6b4999, resource={'id': '175878c3-ae5a-4210-859f-45696c6b4999', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:9a:61:0d', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'd463cd7c-6ca6-4c0e-825e-4d41c90a60f9', 'device_owner': 'network:router_gateway', 'standard_attr_id': 778, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.17'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::348'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T00:59:43Z', 'updated_at': '2026-06-09T00:59:55Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.665302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-175878c3-ae5a-4210-859f-45696c6b4999, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::348/64', '172.24.5.17/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.666110 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=175878c3-ae5a-4210-859f-45696c6b4999, lrouter_port=lrp-175878c3-ae5a-4210-859f-45696c6b4999, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.666110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.666255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:25.666255 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.666255 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:25.666255 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:25.666255 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.666255 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:25.666255 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.666255 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:25.666908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 175878c3-ae5a-4210-859f-45696c6b4999 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.666908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:25.666908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:25.666908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:25.666908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:25.666908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:25.666908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:01:25.666908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:25.666908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.666908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.666908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:25.667153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.667443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:25.667443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.667443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:25.667443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource f612d464-2a07-4d0d-8d1b-99a5d66c807b (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:25.713382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:25.735388 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:25.750756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:25.754467 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-f612d464-2a07-4d0d-8d1b-99a5d66c807b, resource={'id': 'f612d464-2a07-4d0d-8d1b-99a5d66c807b', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:6f:92:50', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '65f33571-0494-499f-be92-d5ac93b22e88', 'device_owner': 'network:router_gateway', 'standard_attr_id': 892, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.139'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::d8'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:00:10Z', 'updated_at': '2026-06-09T01:00:14Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.754628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-f612d464-2a07-4d0d-8d1b-99a5d66c807b, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-65f33571-0494-499f-be92-d5ac93b22e88'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.139/24', '2001:db8::d8/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.754628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=f612d464-2a07-4d0d-8d1b-99a5d66c807b, lrouter_port=lrp-f612d464-2a07-4d0d-8d1b-99a5d66c807b, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.754628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.754628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:25.754628 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.754628 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:25.754628 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:25.754628 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.754628 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:25.754959 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.754959 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:25.755486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource f612d464-2a07-4d0d-8d1b-99a5d66c807b (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.755486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:25.755486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:25.755486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:25.755486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:25.755486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:25.755486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:01:25.755486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:25.755486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.755486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.755486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:25.755843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.756337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:25.756337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.756337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:25.756337 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 4a8b370e-f68f-43ed-939a-0f29e8a361f2 (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:01:25.839007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-4a8b370e-f68f-43ed-939a-0f29e8a361f2, resource={'id': '4a8b370e-f68f-43ed-939a-0f29e8a361f2', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:d4:6d:53', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '7abe82fe-be95-418c-ba05-6ac3294f95e7', 'device_owner': 'network:router_gateway', 'standard_attr_id': 611, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.38'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::264'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T00:57:54Z', 'updated_at': '2026-06-09T00:57:57Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.839161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-4a8b370e-f68f-43ed-939a-0f29e8a361f2, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::264/64', '172.24.5.38/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.839161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=4a8b370e-f68f-43ed-939a-0f29e8a361f2, lrouter_port=lrp-4a8b370e-f68f-43ed-939a-0f29e8a361f2, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:25.839161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.839161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:25.839161 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.839161 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:25.839161 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:25.839161 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.839161 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:25.839507 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.839507 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:25.840369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 4a8b370e-f68f-43ed-939a-0f29e8a361f2 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.840369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:25.840369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:01:25.840369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:01:25.840369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:01:25.840369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:01:25.840369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:01:25.840369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:25.840369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.840369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.840369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:25.840731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:25.841144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:25.841144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:25.841144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:25.841144 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 912b8317-479e-4d76-9666-e9269ce8aeff (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.841144 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(912b8317-479e-4d76-9666-e9269ce8aeff) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.846955 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource fd39a9b0-c5d9-4056-bbb1-9ff5791e1675 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.847316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(fd39a9b0-c5d9-4056-bbb1-9ff5791e1675) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.854323 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource f3078f4b-5614-4e0c-9d43-dc4735628c19 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.854697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(f3078f4b-5614-4e0c-9d43-dc4735628c19) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.860496 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource efd5fce7-30ca-491c-9848-8921a995ba15 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.860836 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(efd5fce7-30ca-491c-9848-8921a995ba15) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.867751 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource e30f2e39-1372-41b1-9c4f-0942e3e13d27 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.868044 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(e30f2e39-1372-41b1-9c4f-0942e3e13d27) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.879149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource d1656dd4-9b2a-414a-8d8f-84585dacdf63 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.879462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(d1656dd4-9b2a-414a-8d8f-84585dacdf63) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.885336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource ad69b4e9-a372-4b07-a2d9-6a110a59dd61 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.885336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(ad69b4e9-a372-4b07-a2d9-6a110a59dd61) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.890566 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource a4f854a7-38fc-47a2-9b08-76e1761932cb (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.890797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(a4f854a7-38fc-47a2-9b08-76e1761932cb) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.899021 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 9f4274e0-50df-4aca-b867-c581eb4571ef (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.899161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(9f4274e0-50df-4aca-b867-c581eb4571ef) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.911432 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 9cb26d68-cd7c-4b9b-95e4-c2682d53259a (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.911702 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(9cb26d68-cd7c-4b9b-95e4-c2682d53259a) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.919033 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 0f5d4d22-2369-4604-8971-426953747ea7 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.919303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(0f5d4d22-2369-4604-8971-426953747ea7) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.933001 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 8a71ec01-434e-4af5-b265-05ca764ec048 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.933414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(8a71ec01-434e-4af5-b265-05ca764ec048) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.940546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 7403119d-d0d8-40a6-a5c5-3ab13dd87991 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.940902 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(7403119d-d0d8-40a6-a5c5-3ab13dd87991) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.949050 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 70399ac3-e09c-4dd6-94fb-3abe59f401e0 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.949335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(70399ac3-e09c-4dd6-94fb-3abe59f401e0) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.954941 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 4d3c82a2-3872-4d38-a74d-c832c5c0a327 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.955288 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(4d3c82a2-3872-4d38-a74d-c832c5c0a327) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.962906 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 3ed3dc93-41bd-4490-8fcf-5f83a6aaae5c (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.962906 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(3ed3dc93-41bd-4490-8fcf-5f83a6aaae5c) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.968639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 2bfde1e8-e307-4c64-85f3-9d39642dab4c (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.968910 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(2bfde1e8-e307-4c64-85f3-9d39642dab4c) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.975052 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 20dab825-ff33-4a34-953e-ca351d1a9427 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.975339 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(20dab825-ff33-4a34-953e-ca351d1a9427) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.982212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 1b65fd72-cd97-4e40-bec7-ac483b6c1a87 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.982552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(1b65fd72-cd97-4e40-bec7-ac483b6c1a87) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.989117 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 1a9a55db-c1e2-44e7-aac6-f2f577e4957e (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:01:25.989433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(1a9a55db-c1e2-44e7-aac6-f2f577e4957e) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:01:25.995524 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_for_inconsistencies finished in 1.837 seconds Jun 09 01:01:25.995725 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:25.995905 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:25.996797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:26.035432 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.035432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.035624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:26.035624 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.035624 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:26.035624 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:26.035624 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.035624 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:26.035624 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.035624 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:26.036372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.036372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:26.036372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:26.036372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:26.036372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.036372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:26.036372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:26.036372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.036372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:26.036372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:26.036372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:26.036372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.036691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:26.037085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:26.037484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.038030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:26.038030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:26.038030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.038030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:26.038030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.038030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:26.038030 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:26.038030 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:26.040300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.040300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.040300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.040300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.040300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.040543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.040543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.040543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.040543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.040543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.041177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:26.041177 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.041177 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:26.041177 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:26.041177 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.041177 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:26.041177 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.041177 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:26.041841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.041841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:26.041841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:26.041841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:26.041841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.041841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:26.041841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:26.041841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.041841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:26.041841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:26.041841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:26.041841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.042264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:26.042569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:26.042828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.043138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:26.043138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:26.043138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.043138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:26.043138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.043138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:26.043138 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:26.043138 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:26.043138 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:01:26.044054 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.044054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.044054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:26.044054 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.044054 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:26.044054 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:26.044054 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.044054 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:26.044054 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.044054 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:26.044584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:26.044855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:26.044855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:26.044855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.044855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:26.044855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:26.044855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.044855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:26.044855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.044855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:26.045460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.045460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:26.045460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:26.045460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:26.045460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.045460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:26.045460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:26.045460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:01:26.045460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:26.045460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:26.045460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.045460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:26.045702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:26.045917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:26.046159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:26.046482 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:26.046698 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:26.047483 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.047483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.047638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:26.047638 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.047638 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:26.047638 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:26.047638 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.047638 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:26.047638 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.047638 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:26.048212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:26.048455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:26.048455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:26.048455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.048455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:26.048455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:26.048455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.048455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:26.048455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.048455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:26.048796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.048796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:26.048796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:26.048796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:26.048796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.048796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:26.048796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:26.048796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:26.048796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:26.048796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.048796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:26.048991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:26.049220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.049220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:26.049220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.049220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.049524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:26.049818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:26.050276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.050276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:26.050276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.050276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:26.050276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:26.050276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:26.050276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:26.050825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.051126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:26.051467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.052057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:26.052057 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.052057 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:26.052057 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:26.052057 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.052057 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:26.052057 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.052057 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:26.052639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.052639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:26.052639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:26.052639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:26.052639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.052639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:26.052639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:26.052639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.052639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:26.052639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:26.052639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:26.052639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.052866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:26.053136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.053136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:26.053297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:26.053670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:26.053966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:26.053966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:26.053966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:30.995909 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:30.996607 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:30.996607 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:30.997134 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:31.037490 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.037978 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:31.038170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.038737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:31.038737 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.038737 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:31.038737 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:31.038737 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.038737 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:31.038737 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.038737 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:31.039624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.039624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:31.039624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:31.039624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:31.039624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.039624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:31.039624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:31.039624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.039624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:31.039624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:31.039624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:31.039624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.039900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:31.040155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.040155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:31.040381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:31.040716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.041089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:31.041089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.041089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:31.041089 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:31.041089 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:31.041439 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.041628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.041800 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.042044 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.042201 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.042367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.042533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.042680 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.043278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:31.043504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.043592 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-702fcf8e-b549-4842-98ab-3fae4b1d895f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.043740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.044576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:31.044576 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.044576 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:31.044576 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:31.044576 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.044576 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:31.044576 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.044576 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:31.045016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.045016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:31.045016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:31.045016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:31.045016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.045016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:31.045016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:31.045016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.045016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:31.045016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:31.045016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:31.045016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.045366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:31.045668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.045668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:31.045737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:31.046322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.046868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:31.046868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.046868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:31.046868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:31.046868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:31.046868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:01:31.046868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:31.046868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.047154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.047154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:31.047154 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.047154 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:31.047154 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:31.047154 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.047154 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:31.047154 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.047154 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:31.047735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:31.048376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:31.048775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:31.049159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:31.049159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:31.049159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:31.049159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:31.049159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:31.049159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.049159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:31.049159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:31.049159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.049159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:31.049159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.049159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:31.049491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:31.049886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:31.050381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.050381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:31.050381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:31.050381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.050381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:31.050381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.050381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:31.050381 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:31.050381 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:31.050381 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:31.050669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.050669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.050669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:31.050669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:31.050669 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.050669 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:31.050669 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:31.050669 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.050669 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:31.050669 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.050669 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:31.050980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:31.054833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:31.055163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:31.055409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:31.055409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:31.055409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:31.055409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:31.055409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.055409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:31.055409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:31.055409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.055409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:31.055409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.055409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:31.055409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:31.055606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.056059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:31.056459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:31.056459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:31.056459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:31.056459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.056459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:31.056459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:31.058261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:31.058949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.059304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:31.059952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:31.061036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:31.061036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:31.061036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:31.061036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:31.061036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:31.061036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:31.061036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:31.061036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:31.061036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:31.061036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:35.996369 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:35.996878 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:35.996878 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:35.997697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:36.041958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:36.050220 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:36.050220 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.050220 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:36.050220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.050220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:36.050220 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.050220 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:36.050220 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:36.050220 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.050220 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:36.050220 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.050220 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:36.051090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.051090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:36.051090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:36.051090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:36.051090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.051090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:36.051090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:36.051090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.051090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:36.051090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:36.051090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:36.051090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.051491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:36.051873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.051873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:36.055212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:36.055609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.056007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:36.056007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.056007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:36.056007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:36.056007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:36.056007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.056007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.056162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.056162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.056162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.056162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.056162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.056274 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.056367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.059366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-702fcf8e-b549-4842-98ab-3fae4b1d895f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.059366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:36.060352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.060836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:36.060836 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.060836 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:36.060836 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:36.060836 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.060836 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:36.060836 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.060836 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:36.061714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.061714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:36.061714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:36.061714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:36.061714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.061714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:36.061714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:36.061714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.061714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:36.061714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:36.061714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:36.061714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.062087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:36.062527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.062527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:36.064245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:36.064715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.065316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:36.065316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.065316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:36.065316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:36.065316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:36.065316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:01:36.065316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.065316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.065522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:36.066097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:36.066542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.066975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:36.076447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:36.076851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:36.077288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:36.077288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:36.077288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.077288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:36.077288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:36.077288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.077288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:36.077288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.077288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:36.077288 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:36.077288 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:36.077288 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:36.077555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.077811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:36.078216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.078216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:36.078216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.078216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:36.078216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:36.078216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:36.078216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.078216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:36.078216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:36.078216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:36.078216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:36.078216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.078431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:36.078695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:36.078958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:36.082478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:36.082709 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:36.082709 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.082709 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:36.082709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.082709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:36.082709 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.082709 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:36.082709 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:36.082709 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.082709 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:36.082709 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.082709 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:36.082994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.082994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:36.082994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:36.082994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:36.082994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.082994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:36.082994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:36.082994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.082994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:36.082994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:36.082994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:36.082994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.085667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:36.086045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:36.086494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:36.087054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:36.087054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:36.087054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:36.087054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:36.087054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:36.087054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:40.996841 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:41.001462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:41.001462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:41.001462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:41.078505 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:41.079326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:41.079326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:41.079326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:41.091705 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.091952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.092623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:41.092623 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.092623 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:41.092623 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:41.092623 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.092623 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:41.092623 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.092623 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:41.093683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.095 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.093683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:41.093683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:41.093683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:41.093683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.093683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:41.093683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:41.093683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.093683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:41.093683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:41.093683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:41.093683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.097008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:41.097444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:41.098520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.099572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:41.099572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:41.099572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.099572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:41.099572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.099572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:41.099572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:41.099572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:41.099572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.099572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.100562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.100562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.100562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.100562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.100562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.100825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.100825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.100825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-702fcf8e-b549-4842-98ab-3fae4b1d895f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.100825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-bfd91133-8761-4fd4-9232-7356b02203ac, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.100825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.100825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:41.100825 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.100825 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:41.100825 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.102132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:41.103305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:41.104007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:41.104862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:41.105886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:41.105886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:41.105886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.105886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:41.105886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:41.105886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.105886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:41.105886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.105886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:41.105886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:41.105886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:41.105886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:41.106669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.107355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:41.109087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.109087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:41.109087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.109087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:41.109087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:41.109087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:41.109087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.109087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:41.109087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:41.109087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:01:41.109087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:41.109087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:41.109473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.109803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.110755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:41.112294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:41.113073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:41.113073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:41.125583 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.125583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.126610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:41.126610 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.126610 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:41.126610 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:41.126610 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.126610 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:41.126610 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.126610 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:41.127479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:41.128116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:41.128116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:41.128116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.128116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:41.128116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:41.128116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.128116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:41.128116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.128116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:41.128753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.022 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.128753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:41.128753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:41.128753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:41.128753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.128753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:41.128753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:41.128753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:41.128753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:41.128753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.128753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:41.129793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:41.130571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.130571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:41.130571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.130571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.130716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:41.131137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:41.131565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.131565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:41.131565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.131565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:41.131565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:41.131565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:41.131565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:41.131941 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.131941 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:41.132426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.132982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:41.132982 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.132982 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:41.132982 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:41.132982 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.132982 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:41.132982 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.132982 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:41.133517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.133517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:41.133517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:41.133517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:41.133517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.133517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:41.133517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:41.133517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.133517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:41.133517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:41.133517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:41.133517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.134027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:41.134553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:41.135039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:41.135574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:41.135574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:41.135574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:41.135574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:41.135574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:41.135574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:45.997358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:45.998338 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:45.998338 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:45.998338 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:46.050705 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.050909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.051567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:46.051567 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.051567 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:46.051567 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:46.051567 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.051567 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:46.051567 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.051567 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:46.052786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.052786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:46.052786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:46.052786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:46.052786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.052786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:46.052786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:46.052786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.052786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:46.052786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:46.052786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:46.052786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.053333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:46.053641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.053641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:46.053980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:46.054438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.054752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:46.054752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.054752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:46.094445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:46.094678 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:46.094800 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:46.095959 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.096169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.096334 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.096483 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.096663 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.096771 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.096921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.097103 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.097259 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-702fcf8e-b549-4842-98ab-3fae4b1d895f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.097661 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-bfd91133-8761-4fd4-9232-7356b02203ac, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.097661 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.097661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.098506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:46.098506 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.098506 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:46.098506 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:46.098506 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.098506 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:46.098506 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.098506 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:46.099051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.099051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:46.099051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:46.099051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:46.099051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.099051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:46.099051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:46.099051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.099051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:46.099051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:46.099051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:46.099051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.099563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:46.099778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.099778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:46.099840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:46.100108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.100412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:46.100412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.100412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:46.101287 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:46.101516 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:46.101603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:46.101694 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:01:46.102112 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.102338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.102847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:46.102847 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.102847 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:46.102847 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:46.102847 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.102847 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:46.102847 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.102847 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:46.103590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:46.104003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:46.104003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:46.104003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.104003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:46.104003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:46.104003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.104003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:46.104003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.104003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:46.104406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.104406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:46.104406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:46.104406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:46.104406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.104406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:46.104406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:46.104406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:01:46.104406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:46.104406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:46.104406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.104406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:46.105131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:46.105418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:46.105640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:46.105946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:46.106169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:46.106169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:46.106377 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.106596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.107065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:46.107065 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.107065 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:46.107065 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:46.107065 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.107065 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:46.107065 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.107065 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:46.107609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:46.107878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:46.107878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:46.107878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.107878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:46.107878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:46.107878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.107878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:46.107878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.107878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:46.108202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.108202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:46.108202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:46.108202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:46.108202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.108202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:46.108202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:46.108202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:46.108202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:46.108202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.108202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:46.108878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:46.109104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:46.109354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:46.109688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:46.109688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:46.109688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:46.109688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:46.109688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.109688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:46.109688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:46.109688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.109688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:46.109688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.109688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:46.131124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:46.131357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:46.131426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:46.131582 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:46.132082 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.132345 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:46.132632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.133181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:46.133181 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.133181 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:46.133181 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:46.133181 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.133181 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:46.133181 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.133181 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:46.133874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.133874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:46.133874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:46.133874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:46.133874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.133874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:46.133874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:46.133874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.133874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:46.133874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:46.133874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:46.133874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:46.134610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.134610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:46.134678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:46.135176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:46.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:46.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:46.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:50.998383 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:50.999017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:50.999017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:50.999531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:51.072903 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.073204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.075 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:51.076211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:51.076748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.077320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:51.077763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:51.078451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:51.078451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:51.078451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.078451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:51.078451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:51.078451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.078451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:51.078451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.078451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:51.094846 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:51.096250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:51.096496 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:51.098042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.098273 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.098399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.098499 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.098598 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.098698 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.099032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.099166 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.099323 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-bfd91133-8761-4fd4-9232-7356b02203ac, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.099451 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.099562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-702fcf8e-b549-4842-98ab-3fae4b1d895f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.099698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:51.101924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:51.102631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.102991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:51.102991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:51.102991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.102991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:51.102991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.102991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:51.104108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:51.105052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.105764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:51.105764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.105764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:51.106030 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:51.106200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:51.107037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:51.107037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:51.107037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:01:51.112458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.112600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.113141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:51.113141 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.113141 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:51.113141 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:51.113141 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.113141 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:51.113141 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.113141 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:51.116232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:51.117091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:51.117559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:51.118091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:51.118091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:51.118091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:51.118091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:51.118091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:51.118091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.118091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:51.118091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:51.118091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.118091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:51.118091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.118091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:51.118592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:51.119157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:51.119659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.119659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:51.119659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:51.119659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.119659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:51.119659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.119659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:51.119659 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:51.119659 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:51.119659 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:51.120733 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.121259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.126217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:51.127207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:51.127791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.128173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:51.128415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:51.128700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:51.129008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:51.129008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.129008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:51.129008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:51.129008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.129008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:51.129008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.129008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:51.134753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:51.134753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:51.134753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:51.134753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:51.134753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.134753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:51.138046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.138608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:51.138608 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.138608 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:51.138608 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:51.138608 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.138608 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:51.138608 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.138608 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:51.144163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.144163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:51.144163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:51.144163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:51.144163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.144163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:51.144163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:51.144163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.144163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:51.144163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:51.144163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:51.144163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.145285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:51.145666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:51.146123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:51.146631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:51.146631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:51.146631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:51.146631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:51.146631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:51.146631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:55.998783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:55.999135 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:55.999135 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:55.999700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:56.027715 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.027867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.028274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:56.028274 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.028274 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:56.028274 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:56.028274 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.028274 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:56.028274 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.028274 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:56.028877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.028877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:56.028877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:56.028877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:56.028877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.028877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:56.028877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:56.028877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.028877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:56.028877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:56.028877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:56.028877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.029129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:56.029424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.029424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:56.029579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:56.029836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.030250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:56.030250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.030250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:56.096296 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:56.096653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:56.096744 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:56.097579 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.097711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.097810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.097914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.098020 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.098116 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.098232 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.098344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.098443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.098540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.099003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:56.099003 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.099003 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:56.099003 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:56.099003 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.099003 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:56.099003 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.099003 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:56.099674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.099674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:56.099674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:56.099674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:56.099674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.099674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:56.099674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:56.099674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.099674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:56.099674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:56.099674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:56.099674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.100034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:56.100364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:56.100761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.101271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:56.101271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:56.101271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.101271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:56.101271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.101271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:56.106601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:56.106849 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:56.106971 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:56.107067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:01:56.107352 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.107501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.107836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:56.107836 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.107836 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:56.107836 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:56.107836 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.107836 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:56.107836 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.107836 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:56.108372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:56.108720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:56.108720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:56.108720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.108720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:56.108720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:56.108720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.108720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:56.108720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.108720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:56.108908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.108908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:56.108908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:56.108908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:56.108908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.108908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:56.108908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:56.108908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:01:56.108908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:56.108908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:56.108908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.108908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:56.109142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:56.109438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.109438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:56.109438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.109438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:56.109587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:56.109855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:56.110221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.110221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:56.110221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:56.110221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.110221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:56.110221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.110221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:56.117006 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:56.117246 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:56.117334 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:56.117432 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:56.117764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.118010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.118470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:56.118470 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.118470 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:56.118470 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:56.118470 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.118470 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:56.118470 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.118470 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:01:56.119037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:01:56.119369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:56.119369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:01:56.119369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.119369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:01:56.119369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:01:56.119369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.119369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:01:56.119369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.119369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:01:56.119996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.119996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:56.119996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:56.119996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:56.119996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.119996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:56.119996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:56.119996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:56.119996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:56.119996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.119996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:56.120244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:56.120561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.120561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:56.120561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.120561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.120799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:56.121140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:56.121550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.121550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:56.121550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.121550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:01:56.132181 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:01:56.132433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:01:56.132549 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:01:56.132712 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:01:56.132987 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.133200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:01:56.133464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.133866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:01:56.133866 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.133866 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:01:56.133866 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:01:56.133866 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.133866 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:01:56.133866 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.133866 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:01:56.134509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.134509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:01:56.134509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:56.134509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:01:56.134509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.134509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:56.134509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:56.134509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.134509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:56.134509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:56.134509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:01:56.134509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.135273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:01:56.135487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:01:56.135788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:01:56.136145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:01:56.136145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:01:56.136145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:01:56.136145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:01:56.136145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:01:56.136145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:00.999226 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:00.999738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:00.999738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:01.000150 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:01.030172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.030400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.031248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:01.031248 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.031248 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:01.031248 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:01.031248 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.031248 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:01.031248 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.031248 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:01.031578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.031578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:01.031578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:01.031578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:01.031578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.031578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:01.031578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:01.031578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.031578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:01.031578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:01.031578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:01.031578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.031835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:01.032081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.032081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:01.032273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:01.032596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.032923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:01.032923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.032923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:01.096759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:01.097063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:01.097153 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:01.098278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.098278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.098463 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.098463 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.098543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.098659 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.098758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.098856 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.098967 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-639a20b9-5f39-4f38-8396-d47ebfd69416, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.099064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.099530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:01.099530 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.099530 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:01.099530 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:01.099530 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.099530 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:01.099530 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.099530 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:01.100203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.100203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:01.100203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:01.100203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:01.100203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.100203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:01.100203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:01.100203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.100203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:01.100203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:01.100203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:01.100203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.100474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:01.100733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.100733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:01.100868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:01.101173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.101578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:01.101578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.101578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:01.106888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:01.107091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:01.107177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:01.107300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:02:01.107576 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.107724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.108092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:01.108092 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.108092 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:01.108092 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:01.108092 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.108092 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:01.108092 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.108092 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:01.108656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:01.109206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.109206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:01.109206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:01.109206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:01.109206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.109206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:01.109206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:01.109206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:02:01.109206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:01.109206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:01.109206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.109206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:01.109506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:01.109916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:01.110231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:01.110563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:01.110563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:01.110563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:01.110563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:01.110563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:01.110563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.110563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:01.110563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:01.110563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.110563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:01.110563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.110563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:01.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:01.117490 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:01.117578 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:01.117668 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:01.117914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.118079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.118467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:01.118467 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.118467 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:01.118467 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:01.118467 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.118467 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:01.118467 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.118467 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:01.118987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:01.119267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:01.119267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:01.119267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.119267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:01.119267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:01.119267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.119267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:01.119267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.119267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:01.119563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.119563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:01.119563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:01.119563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:01.119563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.119563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:01.119563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:01.119563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:01.119563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:01.119563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.119563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:01.119781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:01.120022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.120022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:01.120022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.120022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.120267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:01.120640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:01.120949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.120949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:01.120949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.120949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:01.132542 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:01.132801 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:01.132879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:01.133022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:01.133329 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.133504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:01.133744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:01.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:01.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:01.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:01.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:01.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:01.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:01.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:01.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:01.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:01.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:01.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:01.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:01.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.135087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:01.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:01.135501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:01.135810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:01.136150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:01.136150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:01.136150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:05.999717 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:06.000256 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:06.000256 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:06.000923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:06.033307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.033581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.034198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:06.034198 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.034198 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:06.034198 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:06.034198 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.034198 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:06.034198 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.034198 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:06.035142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.035142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:06.035142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:06.035142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:06.035142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.035142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:06.035142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:06.035142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.035142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:06.035142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:06.035142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:06.035142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.035532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:06.035898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.035898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:06.036284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:06.036746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.037305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:06.037305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.037305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:06.097137 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:06.097414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:06.097507 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:06.098457 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.098638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.098752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.098854 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.098995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.099103 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.099204 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.099308 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.099415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.099869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:06.099869 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.099869 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:06.099869 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:06.099869 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.099869 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:06.099869 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.099869 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:06.100613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.100613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:06.100613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:06.100613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:06.100613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.100613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:06.100613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:06.100613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.100613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:06.100613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:06.100613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:06.100613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.100833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:06.101053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.101053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:06.101360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:06.101618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.101964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:06.101964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.101964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:06.107175 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:06.107380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:06.107470 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:06.107559 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:02:06.108091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.108370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.108948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:06.108948 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.108948 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:06.108948 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:06.108948 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.108948 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:06.108948 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.108948 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:06.109586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:06.109838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:06.109838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:06.109838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.109838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:06.109838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:06.109838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.109838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:06.109838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.109838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:06.110144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.110144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:06.110144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:06.110144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:06.110144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.110144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:06.110144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:06.110144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:02:06.110144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:06.110144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:06.110144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.110144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:06.110367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:06.110582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.110582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:06.110582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.110582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:06.110886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:06.111173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:06.111469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.111469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:06.111469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:06.111469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.111469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:06.111469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.111469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:06.117623 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:06.117893 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:06.118010 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:06.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:06.118632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.118921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.119528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:06.119528 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.119528 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:06.119528 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:06.119528 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.119528 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:06.119528 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.119528 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:06.120147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:06.120431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:06.120431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:06.120431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.120431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:06.120431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:06.120431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.120431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:06.120431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.120431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:06.120785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.120785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:06.120785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:06.120785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:06.120785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.120785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:06.120785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:06.120785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:06.120785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:06.120785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.120785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:06.120993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:06.121221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.121221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:06.121221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.121221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:06.121763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:06.122060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.122060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:06.122060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.122060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:06.132920 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:06.133151 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:06.133266 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:06.133405 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:06.133872 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.134205 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:06.134534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.135132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:06.135132 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.135132 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:06.135132 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:06.135132 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.135132 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:06.135132 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.135132 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:06.135831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.135831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:06.135831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:06.135831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:06.135831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.135831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:06.135831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:06.135831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.135831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:06.135831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:06.135831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:06.135831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.136046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:06.136293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.136293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:06.136587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:06.136829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:06.137130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:06.137130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:06.137130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:11.001484 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:11.002026 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:11.002026 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:11.002636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:11.064846 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.065026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.065530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:11.065530 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.065530 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:11.065530 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:11.065530 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.065530 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:11.065530 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.065530 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:11.068009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.064 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.068009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:11.068009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:11.068009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:11.068009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.068009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:11.068009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:11.068009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.068009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:11.068009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:11.068009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:11.068009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.068412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:11.068666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:11.069065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.069583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:11.069583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:11.069583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.069583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:11.069583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.069583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:11.097604 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:11.098596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:11.098596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:11.099512 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.099653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.099760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.099892 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.100012 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.100113 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.100223 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.100334 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.100438 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.100692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.101026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:11.101026 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.101026 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:11.101026 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:11.101026 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.101026 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:11.101026 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.101026 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:11.101983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.101983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:11.101983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:11.101983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:11.101983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.101983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:11.101983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:11.101983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.101983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:11.101983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:11.101983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:11.101983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.102251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:11.103055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.103055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:11.104921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:11.105407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.105811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:11.105811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.105811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:11.107509 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:11.107754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:11.107883 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:11.108044 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:02:11.108520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.108670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.110251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:11.110251 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.110251 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:11.110251 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:11.110251 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.110251 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:11.110251 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.110251 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:11.111086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:11.111493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:11.111493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:11.111493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.111493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:11.111493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:11.111493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.111493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:11.111493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.111493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:11.112029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.112029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:11.112029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:11.112029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:11.112029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.112029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:11.112029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:11.112029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:02:11.112029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:11.112029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:11.112029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.112029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:11.112392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:11.112706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.112706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:11.112706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.112706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:11.120035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:11.120505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:11.120830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.120830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:11.120830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:11.120830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.120830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:11.120830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.120830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:11.120830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:11.121822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:11.121961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:11.122118 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:11.122709 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.122895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.123335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:11.123335 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.123335 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:11.123335 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:11.123335 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.123335 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:11.123335 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.123335 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:11.124181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:11.124623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:11.124623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:11.124623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.124623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:11.124623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:11.124623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.124623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:11.124623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.124623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:11.125241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.125241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:11.125241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:11.125241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:11.125241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.125241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:11.125241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:11.125241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:11.125241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:11.125241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.125241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:11.125564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:11.125924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.125924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:11.125924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.125924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.126516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:11.127048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:11.127580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.127580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:11.127580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.127580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:11.133291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:11.133508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:11.133606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:11.133739 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:11.134552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.134782 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:11.135099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.135586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:11.135586 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.135586 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:11.135586 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:11.135586 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.135586 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:11.135586 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.135586 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:11.136159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.136159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:11.136159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:11.136159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:11.136159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.136159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:11.136159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:11.136159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.136159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:11.136159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:11.136159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:11.136159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.136625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:11.136839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:11.137106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:11.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:11.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:11.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:11.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:11.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:11.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:16.002038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:16.002533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:16.002533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:16.002882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:16.033599 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.033851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.034473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:16.034473 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.034473 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:16.034473 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:16.034473 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.034473 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:16.034473 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.034473 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:16.035226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.035226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:16.035226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:16.035226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:16.035226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.035226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:16.035226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:16.035226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.035226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:16.035226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:16.035226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:16.035226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.035481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:16.035715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.035715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:16.035918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:16.036233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.036580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:16.036580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.036580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:16.098435 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:16.098643 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:16.098726 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:16.099758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.100027 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.100151 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.100312 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.100449 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.100584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.100712 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.100849 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.100984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.101113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.102913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:16.102913 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.102913 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:16.102913 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:16.102913 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.102913 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:16.102913 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.102913 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:16.103612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.103612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:16.103612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:16.103612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:16.103612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.103612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:16.103612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:16.103612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.103612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:16.103612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:16.103612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:16.103612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.103879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:16.104122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.104122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:16.104283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:16.104570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.104848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:16.104848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.104848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:16.108729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:16.110091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:16.110175 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:16.110283 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:02:16.110780 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.111008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.112198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:16.112198 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.112198 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:16.112198 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:16.112198 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.112198 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:16.112198 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.112198 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:16.112779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:16.113054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:16.113054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:16.113054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.113054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:16.113054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:16.113054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.113054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:16.113054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.113054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:16.113395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.113395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:16.113395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:16.113395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:16.113395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.113395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:16.113395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:16.113395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:02:16.113395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:16.113395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:16.113395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.113395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:16.113635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:16.113860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.113860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:16.113860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.113860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:16.114062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:16.114366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:16.114656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.114656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:16.114656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:16.114656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.114656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:16.114656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.114656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:16.121809 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:16.122938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:16.123050 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:16.123157 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:16.123616 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.123895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:16.124910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:16.125342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:16.125342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:16.125342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.125342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:16.125342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:16.125342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.125342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:16.125342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.125342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:16.125717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.125717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:16.125717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:16.125717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:16.125717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.125717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:16.125717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:16.125717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:16.125717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:16.125717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.125717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:16.126034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:16.126412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.126412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:16.126412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.126412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.126803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:16.127309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:16.127783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.127783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:16.127783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.127783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:16.133600 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:16.133771 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:16.133848 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:16.133989 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:16.134294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.134533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:16.134843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.135362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:16.135362 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.135362 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:16.135362 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:16.135362 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.135362 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:16.135362 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.135362 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:16.136151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.136151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:16.136151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:16.136151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:16.136151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.136151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:16.136151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:16.136151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.136151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:16.136151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:16.136151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:16.136151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:16.136859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.136859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:16.137052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:16.137497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:16.138000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:16.138000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:16.138000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:21.002434 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:21.002940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:21.002940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:21.003589 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:21.004853 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=1104,neutron:UPDATE=3,neutron:DELETE=20 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:02:21.047866 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.048064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.048551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:21.048551 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.048551 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:21.048551 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:21.048551 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.048551 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:21.048551 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.048551 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:21.049461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.049461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:21.049461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:21.049461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:21.049461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.049461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:21.049461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:21.049461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.049461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:21.049461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:21.049461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:21.049461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.049823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:21.050091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.050091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:21.050581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:21.050862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.051212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:21.051212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.051212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:21.098915 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:21.099119 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:21.099119 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:21.100013 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.100093 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.100210 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.100320 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.100408 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.100523 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.100610 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.100717 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8910a7d6-bd8b-4f14-9218-1eb03eb60a13, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.100804 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.100893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.101392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:21.101392 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.101392 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:21.101392 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:21.101392 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.101392 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:21.101392 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.101392 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:21.102030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.102030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:21.102030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:21.102030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:21.102030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.102030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:21.102030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:21.102030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.102030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:21.102030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:21.102030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:21.102030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.102340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:21.102539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.102539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:21.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:21.103162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.103542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:21.103542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.103542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:21.110228 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:21.110463 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:21.110578 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:21.111026 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:02:21.111026 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.111164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.111694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:21.111694 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.111694 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:21.111694 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:21.111694 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.111694 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:21.111694 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.111694 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:21.112236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:21.112570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:21.112570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:21.112570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.112570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:21.112570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:21.112570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.112570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:21.112570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.112570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:21.112818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.112818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:21.112818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:21.112818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:21.112818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.112818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:21.112818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:21.112818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:02:21.112818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:21.112818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:21.112818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.112818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:21.113132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:21.113413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.113413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:21.113413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.113413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:21.113832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:21.114278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:21.114747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.114747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:21.114747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:21.114747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.114747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:21.114747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.114747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:21.123160 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:21.123621 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:21.123831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:21.124022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:21.124440 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.124695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.125283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:21.125283 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.125283 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:21.125283 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:21.125283 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.125283 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:21.125283 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.125283 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:21.126069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:21.126585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:21.126585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:21.126585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.126585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:21.126585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:21.126585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.126585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:21.126585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.126585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:21.127105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.127105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:21.127105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:21.127105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:21.127105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.127105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:21.127105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:21.127105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:21.127105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:21.127105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.127105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:21.127528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:21.127914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.127914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:21.127914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.127914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.128453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:21.128851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:21.129227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.129227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:21.129227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.129227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:21.134043 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:21.134451 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:21.134558 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:21.134740 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:21.135077 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.135343 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:21.135628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.136185 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:21.136185 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.136185 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:21.136185 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:21.136185 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.136185 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:21.136185 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.136185 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:21.137073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.137073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:21.137073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:21.137073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:21.137073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.137073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:21.137073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:21.137073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.137073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:21.137073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:21.137073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:21.137073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.137415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:21.137717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.137717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:21.138103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:21.138550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:21.138963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:21.138963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:21.138963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:26.003258 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:26.003858 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:26.003858 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:26.004826 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:26.050957 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.050957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.051710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:26.051710 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.051710 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:26.051710 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:26.051710 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.051710 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:26.051710 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.051710 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:26.055217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.055217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:26.055217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:26.055217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:26.055217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.055217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:26.055217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:26.055217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.055217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:26.055217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:26.055217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:26.055217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.055836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:26.056261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:26.056644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.057158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:26.057158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:26.057158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.057158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:26.057158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.057158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:26.099291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:26.099633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:26.099692 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:26.100700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.100888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.101068 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.101264 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.101403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.101562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.101712 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.101867 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-31f8b7b1-49f4-4311-958d-2e3c14344865, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.102036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.102611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:26.102611 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.102611 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:26.102611 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:26.102611 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.102611 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:26.102611 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.102611 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:26.103843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.103843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:26.103843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:26.103843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:26.103843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.103843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:26.103843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:26.103843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.103843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:26.103843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:26.103843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:26.103843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.104265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:26.104545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.104545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:26.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:26.106723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.108688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:26.108688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.108688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:26.110578 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:26.110860 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:26.111038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:26.111321 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:02:26.111595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.111811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.112389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:26.112389 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.112389 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:26.112389 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:26.112389 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.112389 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:26.112389 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.112389 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:26.113218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:26.113613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:26.113613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:26.113613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.113613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:26.113613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:26.113613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.113613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:26.113613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.113613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:26.115332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.115332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:26.115332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:26.115332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:26.115332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.115332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:26.115332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:26.115332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:02:26.115332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:26.115332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:26.115332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.115332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:26.115655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:26.115985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:26.116385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:26.116934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:26.116934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:26.116934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:26.116934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:26.116934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:26.116934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.116934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:26.116934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:26.116934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.116934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:26.116934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.116934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:26.123595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:26.123907 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:26.124058 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:26.124241 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:26.124801 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.125102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.125746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:26.125746 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.125746 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:26.125746 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:26.125746 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.125746 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:26.125746 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.125746 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:26.127386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:26.127969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:26.127969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:26.127969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.127969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:26.127969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:26.127969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.127969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:26.127969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.127969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:26.130083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.130083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:26.130083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:26.130083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:26.130083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.130083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:26.130083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:26.130083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:26.130083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:26.130083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.130083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:26.131202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:26.131617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.131617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:26.131617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.131617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.132736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:26.133215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:26.133722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.133722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:26.133722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.133722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:26.135675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:26.135938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:26.136090 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:26.136797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:26.137173 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.137463 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:26.137740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:26.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:26.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:26.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:26.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:26.139613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.139613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:26.139613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:26.139613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:26.139613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.139613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:26.139613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:26.139613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.139613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:26.139613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:26.139613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:26.139613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.139915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:26.140737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.140737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:26.141027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:26.141570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:26.142139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:26.142139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:26.142139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:31.005984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:31.005984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:31.005984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:31.005984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:31.066631 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.067028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.063 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:31.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:31.070396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.070824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:31.071530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:31.072152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:31.072152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:31.072152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.072152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:31.072152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:31.072152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.072152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:31.072152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.072152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:31.099684 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:31.100957 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:31.100957 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:31.102036 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.102523 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.102523 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.102523 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.103564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.103753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.103932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.104212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.104507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:31.109388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:31.109972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:31.110734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:31.111514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:31.111514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:31.111514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:31.111514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.111514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:31.111514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:31.111514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.111514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:31.111514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.111514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:31.111972 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:31.112274 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:31.112492 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:31.112706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:02:31.113158 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.113507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.115064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:31.115064 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.115064 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:31.115064 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:31.115064 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.115064 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:31.115064 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.115064 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:31.115384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:31.115804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:31.115804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:31.115804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.115804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:31.115804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:31.115804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.115804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:31.115804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.115804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:31.117025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.117025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:31.117025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:31.117025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:31.117025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.117025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:31.117025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:31.117025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:02:31.117025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:31.117025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:31.117025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.117025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:31.117450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:31.117796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.117796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:31.117796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.117796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:31.119887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:31.120399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:31.120868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.120868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:31.120868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:31.120868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.120868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:31.120868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.120868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:31.124760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:31.125566 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:31.125566 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:31.125566 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:31.125792 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.125992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:31.130399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.130747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:31.131175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.131175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.132583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:31.133087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:31.133528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.133528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:31.133528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.133528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:31.136259 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:31.140902 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:31.140902 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:31.141274 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:31.143482 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.143861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:31.144203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.144968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:31.144968 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.144968 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:31.144968 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:31.144968 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.144968 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:31.144968 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.144968 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:31.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:31.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:31.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:31.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:31.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:31.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:31.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:31.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:31.146235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.146671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:31.147094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.147094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:31.148234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:31.148706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:31.149263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:31.149263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:31.149263 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:36.004135 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:36.004631 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:36.004705 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:36.005338 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:36.037912 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.038115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.038691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:36.038691 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.038691 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:36.038691 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:36.038691 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.038691 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:36.038691 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.038691 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:36.039424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.039424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:36.039424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:36.039424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:36.039424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.039424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:36.039424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:36.039424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.039424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:36.039424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:36.039424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:36.039424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.039843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:36.040267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:36.040696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.041210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:36.041210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:36.041210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.041210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:36.041210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.041210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:36.100267 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:36.100609 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:36.100703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:36.101664 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.101900 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.102145 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.102373 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.102597 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.102783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.102986 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.103256 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.103470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.104085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:36.104085 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.104085 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:36.104085 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:36.104085 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.104085 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:36.104085 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.104085 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:36.104951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.104951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:36.104951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:36.104951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:36.104951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.104951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:36.104951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:36.104951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.104951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:36.104951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:36.104951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:36.104951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.105165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:36.105393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.105393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:36.105693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:36.106102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.106579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:36.106579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.106579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:36.113435 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:36.113733 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:36.113822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:36.113914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:02:36.114473 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.114759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.115394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:36.115394 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.115394 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:36.115394 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:36.115394 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.115394 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:36.115394 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.115394 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:36.116009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:36.116288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:36.116288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:36.116288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.116288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:36.116288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:36.116288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.116288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:36.116288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.116288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:36.116678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.116678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:36.116678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:36.116678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:36.116678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.116678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:36.116678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:36.116678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:02:36.116678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:36.116678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:36.116678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.116678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:36.116876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:36.117101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.117101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:36.117101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.117101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:36.117385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:36.117634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:36.117896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.117896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:36.117896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:36.117896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.117896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:36.117896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.117896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:36.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:36.125618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:36.125694 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:36.125802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:36.126232 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.126479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.127028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:36.127028 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.127028 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:36.127028 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:36.127028 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.127028 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:36.127028 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.127028 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:36.127587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:36.127982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:36.127982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:36.127982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.127982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:36.127982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:36.127982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.127982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:36.127982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.127982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:36.128208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.128208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:36.128208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:36.128208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:36.128208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.128208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:36.128208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:36.128208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:36.128208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:36.128208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.128208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:36.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:36.128827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:36.129301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:36.129777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:36.129777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:36.129777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:36.129777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:36.129777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.129777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:36.129777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:36.129777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.129777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:36.129777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.129777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:36.141256 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:36.141555 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:36.141637 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:36.141791 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:36.142216 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.142462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:36.142721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.143691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:36.143691 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.143691 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:36.143691 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:36.143691 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.143691 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:36.143691 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.143691 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:36.144314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.144314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:36.144314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:36.144314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:36.144314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.144314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:36.144314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:36.144314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.144314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:36.144314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:36.144314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:36.144314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.144530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:36.144717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.144717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:36.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:36.145242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:36.145541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:36.145541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:36.145541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:41.005380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:41.005894 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:41.005962 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:41.006756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:41.041463 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.041702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.042303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:41.042303 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.042303 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:41.042303 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:41.042303 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.042303 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:41.042303 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.042303 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:41.042990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.042990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:41.042990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:41.042990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:41.042990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.042990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:41.042990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:41.042990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.042990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:41.042990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:41.042990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:41.042990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.043420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:41.043827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:41.044295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.044649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:41.044649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:41.044649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.044649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:41.044649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.044649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:41.100736 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:41.101115 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:41.101227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:41.102314 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.102576 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.102865 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.103093 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.103345 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.103513 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.103715 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.104028 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.104259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.104802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:41.104802 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.104802 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:41.104802 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:41.104802 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.104802 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:41.104802 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.104802 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:41.105713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.105713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:41.105713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:41.105713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:41.105713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.105713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:41.105713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:41.105713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.105713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:41.105713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:41.105713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:41.105713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.106143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:41.106606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.106606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:41.106755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:41.107247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.107771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:41.107771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.107771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:41.113860 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:41.114342 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:41.114471 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:41.114597 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:02:41.114992 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.115175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.115686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:41.115686 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.115686 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:41.115686 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:41.115686 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.115686 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:41.115686 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.115686 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:41.116548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:41.116885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:41.116885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:41.116885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.116885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:41.116885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:41.116885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.116885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:41.116885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.116885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:41.117396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.117396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:41.117396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:41.117396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:41.117396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.117396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:41.117396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:41.117396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:02:41.117396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:41.117396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:41.117396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.117396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:41.117669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:41.117987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.117987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:41.117987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.117987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:41.118430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:41.118751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:41.119159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.119159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:41.119159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:41.119159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.119159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:41.119159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.119159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:41.125714 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:41.126151 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:41.126292 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:41.126426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:41.126995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.127370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.128023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:41.128023 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.128023 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:41.128023 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:41.128023 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.128023 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:41.128023 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.128023 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:41.128868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:41.129200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:41.129200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:41.129200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.129200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:41.129200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:41.129200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.129200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:41.129200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.129200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:41.129789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.129789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:41.129789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:41.129789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:41.129789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.129789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:41.129789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:41.129789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:41.129789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:41.129789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.129789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:41.130032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:41.130281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.130281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:41.130281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.130281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:41.131182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:41.131526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.131526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:41.131526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.131526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:41.141704 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:41.142107 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:41.142251 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:41.142429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:41.142821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.143048 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:41.143298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.143817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:41.143817 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.143817 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:41.143817 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:41.143817 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.143817 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:41.143817 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.143817 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:41.144813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.144813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:41.144813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:41.144813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:41.144813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.144813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:41.144813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:41.144813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.144813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:41.144813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:41.144813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:41.144813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.145149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:41.145425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.145425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:41.145830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:41.146149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:41.146512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:41.146512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:41.146512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:46.005806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:46.007923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:46.007923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:46.007923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:46.039711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.039962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.040748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:46.040748 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.040748 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:46.040748 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:46.040748 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.040748 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:46.040748 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.040748 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:46.041624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.041624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:46.041624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:46.041624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:46.041624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.041624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:46.041624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:46.041624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.041624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:46.041624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:46.041624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:46.041624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.043004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:46.043233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.043233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:46.043847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:46.044696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.045636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:46.045636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.045636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:46.101230 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:46.101668 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:46.101862 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:46.102741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.102877 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.104612 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.104612 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.104612 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.104612 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.104612 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.104770 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.104770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.105151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:46.105151 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.105151 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:46.105151 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:46.105151 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.105151 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:46.105151 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.105151 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:46.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:46.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:46.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:46.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:46.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:46.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:46.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:46.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:46.106148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.106848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:46.108663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.108663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:46.109295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:46.110124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.110686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:46.110686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.110686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:46.114220 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:46.114452 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:46.115068 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:46.115068 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:02:46.115068 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.115256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.116103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:46.116103 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.116103 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:46.116103 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:46.116103 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.116103 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:46.116103 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.116103 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:46.116606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:46.118677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:46.118922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:46.119124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:46.119124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:46.119124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:46.119124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:46.119124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:46.119124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.119124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:46.119124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:46.119124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.119124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:46.119124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.119124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:46.119513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:46.119830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:46.120131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.120131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:46.120131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:46.120131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.120131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:46.120131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.120131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:46.126758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:46.126758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:46.126758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:46.126758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:46.127720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.127720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.127720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:46.127720 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.127720 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:46.127720 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:46.127720 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.127720 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:46.127720 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.127720 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:46.130206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:46.130473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:46.130712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:46.130906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:46.130906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:46.130906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:46.130906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:46.130906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.130906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:46.130906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:46.130906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.130906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:46.130906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.130906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:46.130906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:46.131121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.131469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:46.142140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:46.142452 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:46.142561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:46.142715 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:46.143294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.143294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:46.143574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.144061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:46.144061 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.144061 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:46.144061 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:46.144061 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.144061 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:46.144061 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.144061 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:46.144675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.144675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:46.144675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:46.144675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:46.144675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.144675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:46.144675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:46.144675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.144675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:46.144675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:46.144675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:46.144675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:46.145540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:46.145896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:46.146382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:46.146382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:46.146382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:46.146382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:46.146382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:46.146382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:51.006657 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:51.008754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:51.008754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:51.008754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:51.043458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.043667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:51.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:51.045832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.046078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:51.046349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:51.046690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:51.046690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:51.046690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.046690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:51.046690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:51.046690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.046690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:51.046690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.046690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:51.102494 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:51.103957 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:51.103957 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:51.105508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.105728 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.105821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.105967 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.106098 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.106248 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.106487 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.106873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.107040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:51.112222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:51.112914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.113312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:51.113744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:51.114312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:51.114312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:51.114312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:51.114312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.114312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:51.114312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:51.114312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.114312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:51.114312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.114312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:51.114593 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:51.122096 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:51.122096 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:51.122096 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:02:51.122096 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.122096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.122096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:51.122096 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.122096 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:51.122096 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:51.122764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.124487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:51.124942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.125371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:51.125848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:51.126348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:51.126918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:51.126918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:51.126918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.126918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:51.126918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:51.126918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.126918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:51.126918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.126918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:51.126918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:51.126918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:51.126918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:51.128644 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:51.128644 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.128644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.129329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:51.129329 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.129329 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:51.129329 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:51.129329 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.129329 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:51.129329 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.129329 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:51.130301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:51.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:51.132595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:51.134094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:51.134094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:51.134094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:51.134094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:51.134094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.134094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:51.134094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:51.134094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.134094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:51.134094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.134094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:51.137736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:51.138322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.138322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:51.138322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.138322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:51.142556 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:51.142753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:51.142845 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:51.142995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:51.143280 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.143458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:51.143647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:51.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:51.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:51.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:51.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.144037 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:51.144608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.144608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:51.144608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:51.144608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:51.144608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.144608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:51.144608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:51.144608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.144608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:51.144608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:51.144608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:51.144608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.144961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:51.145290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:51.145682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:51.146155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:51.146155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:51.146155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:51.146155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:51.146155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:51.146155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:56.007486 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:56.008037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:56.008037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:56.008521 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:56.044953 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.045174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.045693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:56.045693 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.045693 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:56.045693 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:56.045693 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.045693 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:56.045693 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.045693 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:56.046597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.046597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:56.046597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:56.046597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:56.046597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.046597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:56.046597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:56.046597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.046597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:56.046597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:56.046597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:56.046597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.047256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:56.048078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:56.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.048995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:56.048995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:56.048995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.048995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:56.048995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.048995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:56.103239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:56.103427 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:56.103546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:56.104573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.104778 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.104880 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.105199 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.105324 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.105457 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.105564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.105659 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.105751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.106223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:56.106223 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.106223 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:56.106223 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:56.106223 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.106223 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:56.106223 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.106223 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:56.107112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.107112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:56.107112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:56.107112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:56.107112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.107112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:56.107112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:56.107112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.107112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:56.107112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:56.107112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:56.107112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:56.107764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.107764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:56.107802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:56.108065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.108556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:56.108556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.108556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:56.116349 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:56.116572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:56.116918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:56.117083 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:02:56.117457 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.117618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.118068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:56.118068 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.118068 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:56.118068 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:56.118068 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.118068 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:56.118068 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.118068 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:56.119073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:56.119376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:56.119376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:56.119376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.119376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:56.119376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:56.119376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.119376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:56.119376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.119376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:56.119905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.119905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:56.119905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:56.119905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:56.119905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.119905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:56.119905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:56.119905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:02:56.119905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:56.119905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:56.119905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.119905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:56.120404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:56.120612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.120612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:56.120612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.120612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:56.120968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:56.121547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:56.122162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.122162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:56.122162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:56.122162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.122162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:56.122162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.122162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:56.126882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:56.127139 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:56.127291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:56.127450 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:56.127945 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.128157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.128823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:56.128823 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.128823 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:56.128823 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:56.128823 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.128823 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:56.128823 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.128823 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:02:56.129921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:02:56.130212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:56.130212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:02:56.130212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.130212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:02:56.130212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:02:56.130212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.130212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:02:56.130212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.130212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:02:56.131227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.131227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:56.131227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:56.131227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:56.131227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.131227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:56.131227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:56.131227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:56.131227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:56.131227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.131227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:56.131436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:56.131645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.131645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:56.131645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.131645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.132679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:56.133131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:56.133585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.133585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:56.133585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.133585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:02:56.142961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:02:56.143249 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:02:56.143340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:02:56.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:02:56.147531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.147754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:02:56.148000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.148470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:02:56.148470 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.148470 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:02:56.148470 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:02:56.148470 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.148470 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:02:56.148470 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.148470 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:02:56.149457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.149457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:02:56.149457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:56.149457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:02:56.149457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.149457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:56.149457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:56.149457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.149457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:56.149457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:56.149457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:02:56.149457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.149680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:02:56.149869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.149869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:02:56.150622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:02:56.150880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:02:56.151210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:02:56.151210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:02:56.151210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:01.007892 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:01.008358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:01.008406 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:01.009056 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:01.045253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.045515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.046030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:01.046030 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.046030 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:01.046030 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:01.046030 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.046030 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:01.046030 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.046030 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:01.046789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.046789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:01.046789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:01.046789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:01.046789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.046789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:01.046789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:01.046789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.046789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:01.046789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:01.046789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:01.046789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.047267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:01.047642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:01.048450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.049100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:01.049100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:01.049100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.049100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:01.049100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.049100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:01.103587 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:01.103857 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:01.104005 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:01.105372 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.105573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.105703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.105810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.106075 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.106356 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.106356 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.106433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.106568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.107266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:01.107266 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.107266 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:01.107266 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:01.107266 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.107266 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:01.107266 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.107266 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:01.108964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.109439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:01.109950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.109950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:01.110146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:01.110597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.111095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:01.111095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.111095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:01.116737 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:01.116954 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:01.117071 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:01.117205 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:03:01.117692 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.117867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.118583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:01.118583 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.118583 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:01.118583 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:01.118583 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.118583 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:01.118583 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.118583 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:01.119349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:01.119770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:01.119770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:01.119770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.119770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:01.119770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:01.119770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.119770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:01.119770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.119770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:01.120097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.120097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:01.120097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:01.120097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:01.120097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.120097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:01.120097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:01.120097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:03:01.120097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:01.120097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:01.120097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.120097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:01.120365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:01.120599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.120599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:01.120599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.120599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:01.121029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:01.121356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:01.121660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.121660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:01.121660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:01.121660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.121660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:01.121660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.121660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:01.127317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:01.127552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:01.127637 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:01.127747 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:01.128122 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.128444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.128930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:01.128930 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.128930 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:01.128930 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:01.128930 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.128930 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:01.128930 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.128930 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:01.129676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:01.129987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:01.129987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:01.129987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.129987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:01.129987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:01.129987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.129987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:01.129987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.129987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:01.130342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.130342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:01.130342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:01.130342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:01.130342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.130342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:01.130342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:01.130342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:01.130342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:01.130342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.130342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:01.130703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:01.131022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.131022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:01.131022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.131022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.131203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:01.131630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:01.131988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.131988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:01.131988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.131988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:01.143321 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:01.143550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:01.143656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:01.143795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:01.144118 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.144344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:01.144542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.144952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:01.144952 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.144952 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:01.144952 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:01.144952 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.144952 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:01.144952 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.144952 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:01.145562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.145562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:01.145562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:01.145562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:01.145562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.145562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:01.145562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:01.145562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.145562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:01.145562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:01.145562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:01.145562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.145933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:01.146291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:01.146941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:01.147468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:01.147468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:01.147468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:01.147468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:01.147468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:01.147468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:06.008395 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:06.009098 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:06.009098 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:06.009605 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:06.040433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.040622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.041060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:06.041060 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.041060 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:06.041060 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:06.041060 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.041060 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:06.041060 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.041060 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:06.041708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.041708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:06.041708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:06.041708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:06.041708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.041708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:06.041708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:06.041708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.041708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:06.041708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:06.041708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:06.041708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.042420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:06.042726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:06.043231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.043661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:06.043661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:06.043661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.043661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:06.043661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.043661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:06.104039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:06.104352 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:06.104486 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:06.106654 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.106836 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.106993 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.107167 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.107307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.107446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.107573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7abe82fe-be95-418c-ba05-6ac3294f95e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.107703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.107829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.108391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:06.108391 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.108391 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:06.108391 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:06.108391 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.108391 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:06.108391 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.108391 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:06.109102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.109102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:06.109102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:06.109102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:06.109102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.109102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:06.109102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:06.109102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.109102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:06.109102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:06.109102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:06.109102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.109558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:06.110027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:06.110523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.110944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:06.110944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:06.110944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.110944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:06.110944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.110944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:06.117120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:06.117430 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:06.117540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:06.117675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:03:06.118149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.118396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.118921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:06.118921 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.118921 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:06.118921 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:06.118921 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.118921 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:06.118921 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.118921 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:06.119778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:06.120118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:06.120118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:06.120118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.120118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:06.120118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:06.120118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.120118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:06.120118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.120118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:06.120739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.120739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:06.120739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:06.120739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:06.120739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.120739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:06.120739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:06.120739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:03:06.120739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:06.120739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:06.120739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.120739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:06.121359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:06.121618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.121618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:06.121618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.121618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:06.121719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:06.122257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:06.122823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.122823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:06.122823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:06.122823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.122823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:06.122823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.122823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:06.127723 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:06.128149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:06.128269 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:06.128438 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:06.129867 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.130096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.130604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:06.130604 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.130604 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:06.130604 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:06.130604 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.130604 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:06.130604 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.130604 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:06.131489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:06.131826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:06.131826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:06.131826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.131826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:06.131826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:06.131826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.131826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:06.131826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.131826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:06.132407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.132407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:06.132407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:06.132407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:06.132407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.132407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:06.132407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:06.132407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:06.132407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:06.132407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.132407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:06.132662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:06.132924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.132924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:06.132924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.132924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.133548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:06.133894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:06.134274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.134274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:06.134274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.134274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:06.143810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:06.144374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:06.144466 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:06.144665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:06.145044 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.145333 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:06.145626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.146182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:06.146182 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.146182 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:06.146182 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:06.146182 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.146182 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:06.146182 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.146182 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:06.146915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.146915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:06.146915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:06.146915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:06.146915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.146915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:06.146915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:06.146915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.146915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:06.146915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:06.146915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:06.146915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.147350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:06.147711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:06.148138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:06.148657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:06.148657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:06.148657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:06.148657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:06.148657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:06.148657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:11.009530 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:11.010404 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:11.010404 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:11.010688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:11.044723 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:11.045012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.045609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:11.045609 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.045609 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:11.045609 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:11.045609 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.045609 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:11.045609 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.045609 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:11.046598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.046598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:11.046598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:11.046598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:11.046598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.046598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:11.046598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:11.046598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.046598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:11.046598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:11.046598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:11.046598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.047031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:11.047398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.047398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:11.047734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:11.048080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.048567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:11.048567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.048567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:11.104475 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:11.106614 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:11.106737 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:11.108298 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:11.108298 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:11.108691 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:11.108897 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:11.109233 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:11.109233 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:11.109769 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:11.109909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.110766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:11.110766 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.110766 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:11.110766 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:11.110766 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.110766 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:11.110766 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.110766 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:11.111339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.111339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:11.111339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:11.111339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:11.111339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.111339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:11.111339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:11.111339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.111339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:11.111339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:11.111339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:11.111339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.111784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:11.112331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.112331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:11.112574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:11.113085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.113438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:11.113438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.113438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:11.117580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:11.117905 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:11.118117 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:11.118307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:03:11.118675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:11.119248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.119814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:11.119814 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.119814 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:11.119814 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:11.119814 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.119814 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:11.119814 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.119814 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:11.120743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:11.121040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:11.121040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:11.121040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.121040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:11.121040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:11.121040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.121040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:11.121040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.121040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:11.121685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.121685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:11.121685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:11.121685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:11.121685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.121685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:11.121685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:11.121685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:03:11.121685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:11.121685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:11.121685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.121685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:11.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:11.122154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.122154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:11.122154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.122154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:11.122785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:11.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:11.123459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.123459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:11.123459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:11.123459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.123459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:11.123459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.123459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:11.128181 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:11.128462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:11.128591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:11.128710 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:11.129040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:11.129286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.129722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:11.129722 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.129722 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:11.129722 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:11.129722 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.129722 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:11.129722 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.129722 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:11.130267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:11.130530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:11.130530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:11.130530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.130530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:11.130530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:11.130530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.130530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:11.130530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.130530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:11.131025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.131025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:11.131025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:11.131025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:11.131025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.131025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:11.131025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:11.131025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:11.131025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:11.131025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.131025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:11.131251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:11.131540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.131540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:11.131540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.131540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.131681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:11.131992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:11.132382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.132382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:11.132382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.132382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:11.144437 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:11.144700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:11.144700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:11.144847 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:11.145143 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:11.145368 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:11.145535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.145876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:11.145876 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.145876 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:11.145876 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:11.145876 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.145876 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:11.145876 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.145876 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:11.146453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.146453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:11.146453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:11.146453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:11.146453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.146453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:11.146453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:11.146453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.146453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:11.146453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:11.146453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:11.146453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.146682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:11.146880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.146880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:11.147198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:11.147496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:11.147907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:11.147907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:11.147907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:16.010281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:16.010802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:16.010802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:16.011448 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:16.053933 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:16.054121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.054599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:16.054599 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.054599 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:16.054599 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:16.054599 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.054599 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:16.054599 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.054599 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:16.055608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.055608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:16.055608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:16.055608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:16.055608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.055608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:16.055608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:16.055608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.055608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:16.055608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:16.055608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:16.055608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.056012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:16.056525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.056525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:16.056743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:16.057174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.057695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:16.057695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.057695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:16.107079 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:16.107378 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:16.107492 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:16.108472 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:16.108628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:16.108753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:16.108899 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:16.109027 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:16.109143 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:16.109284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:16.109403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.109899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:16.109899 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.109899 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:16.109899 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:16.109899 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.109899 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:16.109899 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.109899 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:16.111145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.111145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:16.111145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:16.111145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:16.111145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.111145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:16.111145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:16.111145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.111145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:16.111145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:16.111145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:16.111145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.111393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:16.111593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.111593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:16.111833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:16.112448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.113059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:16.113059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.113059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:16.117872 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:16.118144 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:16.118307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:16.118458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:03:16.118806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:16.119038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.119584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:16.119584 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.119584 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:16.119584 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:16.119584 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.119584 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:16.119584 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.119584 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:16.120334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:16.121180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:16.121558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:16.121888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:16.121888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:16.121888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:16.121888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:16.121888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:16.121888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.121888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:16.121888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:16.121888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.121888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:16.121888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.121888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:16.122570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:16.123546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.124428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:16.128601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:16.128866 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:16.129030 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:16.129183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:16.129574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:16.129845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.130414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:16.130414 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.130414 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:16.130414 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:16.130414 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.130414 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:16.130414 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.130414 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:16.131075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:16.131486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:16.131486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:16.131486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.131486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:16.131486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:16.131486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.131486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:16.131486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.131486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:16.132149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.132149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:16.132149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:16.132149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:16.132149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.132149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:16.132149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:16.132149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:16.132149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:16.132149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.132149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:16.132705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:16.133523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:16.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:16.134213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:16.134213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:16.134213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:16.134213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:16.134213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.134213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:16.134213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:16.134213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.134213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:16.134213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.134213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:16.144773 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:16.145017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:16.145101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:16.145294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:16.146598 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:16.146799 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:16.147021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.147523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:16.147523 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.147523 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:16.147523 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:16.147523 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.147523 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:16.147523 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.147523 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:16.148277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.148277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:16.148277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:16.148277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:16.148277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.148277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:16.148277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:16.148277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.148277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:16.148277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:16.148277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:16.148277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.148579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:16.148859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.148859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:16.149033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:16.150385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:16.150991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:16.150991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:16.150991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:21.010632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:21.011371 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:21.012011 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:21.012656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:21.018695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=241 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:03:21.055142 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:21.055341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.056069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:21.056069 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.056069 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:21.056069 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:21.056069 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.056069 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:21.056069 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.056069 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:21.057319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.057319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:21.057319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:21.057319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:21.057319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.057319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:21.057319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:21.057319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.057319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:21.057319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:21.057319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:21.057319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.057837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:21.058325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.058325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:21.058788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:21.059350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.059960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:21.059960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.059960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:21.107618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:21.108888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:21.109022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:21.109928 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:21.110136 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:21.110292 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:21.110433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:21.110558 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:21.110722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:21.110856 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:21.111008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.112510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:21.112510 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.112510 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:21.112510 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:21.112510 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.112510 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:21.112510 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.112510 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:21.113492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.113492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:21.113492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:21.113492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:21.113492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.113492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:21.113492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:21.113492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.113492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:21.113492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:21.113492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:21.113492 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.113781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:21.114005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.114005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:21.117104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:21.117588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.117916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:21.117916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.117916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:21.118159 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:21.119325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:21.119398 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:21.119511 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:03:21.119844 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:21.120076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.120889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:21.120889 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.120889 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:21.120889 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:21.120889 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.120889 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:21.120889 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.120889 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:21.121437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:21.121733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:21.121733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:21.121733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.121733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:21.121733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:21.121733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.121733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:21.121733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.121733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:21.121983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.121983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:21.121983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:21.121983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:21.121983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.121983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:21.121983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:21.121983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:03:21.121983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:21.121983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:21.121983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.121983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:21.122398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:21.122744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:21.123117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:21.123523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:21.123523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:21.123523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:21.123523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:21.123523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:21.123523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.123523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:21.123523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:21.123523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.123523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:21.123523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.123523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:21.129048 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:21.129368 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:21.129485 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:21.129596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:21.129890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:21.130089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.130595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:21.130595 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.130595 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:21.130595 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:21.130595 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.130595 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:21.130595 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.130595 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:21.131145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:21.131444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:21.131444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:21.131444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.131444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:21.131444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:21.131444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.131444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:21.131444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.131444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:21.131772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.131772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:21.131772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:21.131772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:21.131772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.131772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:21.131772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:21.131772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:21.131772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:21.131772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.131772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:21.132113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:21.132445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.132445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:21.132445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.132445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.132545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:21.132897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:21.133366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.133366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:21.133366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.133366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:21.145237 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:21.145604 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:21.145715 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:21.145866 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:21.146200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:21.146407 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:21.146611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.147175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:21.147175 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.147175 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:21.147175 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:21.147175 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.147175 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:21.147175 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.147175 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:21.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:21.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:21.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:21.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:21.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:21.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:21.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:21.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:21.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.148261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:21.148517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.148517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:21.149019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:21.149328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:21.149687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:21.149687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:21.149687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:26.011115 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:26.011742 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:26.011875 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:26.012595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:26.051423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:26.051716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.052572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:26.052572 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.052572 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:26.052572 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:26.052572 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.052572 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:26.052572 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.052572 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:26.053497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.053497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:26.053497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:26.053497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:26.053497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.053497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:26.053497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:26.053497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.053497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:26.053497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:26.053497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:26.053497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.053953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:26.054364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.054364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:26.054639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:26.055161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.055690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:26.055690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.055690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:26.111080 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:26.111174 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:26.113134 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:26.113436 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:26.113580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:26.113655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:26.115126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d463cd7c-6ca6-4c0e-825e-4d41c90a60f9, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:26.115126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:26.115126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:26.115126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.115668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:26.115668 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.115668 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:26.115668 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:26.115668 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.115668 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:26.115668 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.115668 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:26.117081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.117081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:26.117081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:26.117081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:26.117081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.117081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:26.117081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:26.117081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.117081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:26.117081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:26.117081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:26.117081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.117440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:26.117721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.117721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:26.119825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:26.120327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.120831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:26.120831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.120831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:26.121342 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:26.121342 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:26.121511 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:26.121879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:03:26.124650 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:26.124899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.125623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:26.125623 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.125623 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:26.125623 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:26.125623 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.125623 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:26.125623 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.125623 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:26.126798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:26.127159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:26.127159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:26.127159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.127159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:26.127159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:26.127159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.127159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:26.127159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.127159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:26.128684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.128684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:26.128684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:26.128684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:26.128684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.128684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:26.128684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:26.128684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:03:26.128684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:26.128684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:26.128684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.128684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:26.129069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:26.129464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.129464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:26.129464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.129464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:26.130981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:26.131560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:26.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:26.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:26.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:26.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:26.132159 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:26.132568 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:26.132737 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:26.132961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:26.133428 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:26.133762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:26.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:26.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:26.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:26.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:26.135431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:26.135783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:26.135783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:26.135783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.135783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:26.135783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:26.135783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.135783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:26.135783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.135783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:26.136597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.136597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:26.136597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:26.136597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:26.136597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.136597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:26.136597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:26.136597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:26.136597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:26.136597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.136597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:26.136888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:26.137356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.137356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:26.137356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.137356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.138008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:26.138393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:26.138696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.138696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:26.138696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.138696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:26.145724 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:26.148393 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:26.148763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:26.148763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:26.150374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:26.150625 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:26.151097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.151338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:26.151338 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.151338 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:26.151338 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:26.151338 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.151338 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:26.151338 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.151338 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:26.152576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.152576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:26.152576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:26.152576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:26.152576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.152576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:26.152576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:26.152576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.152576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:26.152576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:26.152576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:26.152576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.153057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:26.153417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.153417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:26.154556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:26.155062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:26.155565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:26.155565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:26.155565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:31.011940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:31.013399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:31.013541 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:31.014127 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:31.052948 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:31.053278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.053797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:31.053797 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.053797 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:31.053797 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:31.053797 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.053797 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:31.053797 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.053797 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:31.054781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.054781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:31.054781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:31.054781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:31.054781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.054781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:31.054781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:31.054781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.054781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:31.054781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:31.054781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:31.054781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.055058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:31.055311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.055311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:31.056123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:31.056441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.056777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:31.056777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.056777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:31.111137 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:31.112683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:31.112683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:31.113346 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:31.113983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:31.114280 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:31.114537 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:31.114813 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:31.115075 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:31.115366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.116081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:31.116081 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.116081 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:31.116081 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:31.116081 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.116081 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:31.116081 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.116081 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:31.117346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.117346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:31.117346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:31.117346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:31.117346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.117346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:31.117346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:31.117346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.117346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:31.117346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:31.117346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:31.117346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.117994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:31.118421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.118421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:31.120247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:31.120764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.121237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:31.121237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.121237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:31.121648 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:31.121920 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:31.122499 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:31.122681 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:03:31.123092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:31.123372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.124281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:31.124281 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.124281 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:31.124281 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:31.124281 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.124281 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:31.124281 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.124281 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:31.125368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:31.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:31.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:31.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:31.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:31.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:31.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:31.128015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.128015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:31.128015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:31.128015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:31.128015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.128015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:31.128015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:31.128015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:03:31.128015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:31.128015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:31.128015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.128015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:31.128370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:31.128678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:31.128977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:31.129412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:31.129412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:31.129412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:31.129412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:31.129412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:31.129412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.129412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:31.129412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:31.129412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.129412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:31.129412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.129412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:31.132653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:31.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:31.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:31.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:31.133761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:31.134017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:31.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:31.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:31.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:31.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:31.135316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:31.135695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:31.135695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:31.135695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.135695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:31.135695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:31.135695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.135695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:31.135695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.135695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:31.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:31.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:31.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:31.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:31.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:31.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:31.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:31.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:31.138415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:31.138673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:31.138947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:31.139293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:31.139293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:31.139293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:31.139293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:31.139293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.139293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:31.139293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:31.139293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.139293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:31.139293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.139293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:31.148504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:31.149214 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:31.149214 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:31.149214 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:31.149528 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:31.149778 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:31.150051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.150767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:31.150767 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.150767 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:31.150767 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:31.150767 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.150767 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:31.150767 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.150767 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:31.152946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.152946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:31.152946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:31.152946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:31.152946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.152946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:31.152946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:31.152946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.152946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:31.152946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:31.152946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:31.152946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.153260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:31.153481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:31.153776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:31.154306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:31.154306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:31.154306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:31.154306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:31.154306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:31.154306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:36.013385 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:36.015157 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:36.015157 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:36.015530 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:36.070310 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:36.070422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:36.071604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.071604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:36.071604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:36.071604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:36.071604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.071604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:36.071604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:36.071604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.071604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:36.071604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:36.071604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:36.071604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.071965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:36.072376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:36.072792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.073405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:36.073405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:36.073405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.073405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:36.073405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.073405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:36.111748 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:36.111996 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:36.112101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:36.112808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:36.112935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:36.113049 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:36.113146 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:36.113281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:36.115212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:36.115836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:36.116290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:36.116290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:36.116290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.116290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:36.116290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:36.116290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.116290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:36.116290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.116290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:36.116290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.116290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:36.116290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:36.116704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.117331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:36.122074 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:36.122955 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:36.123049 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:36.123141 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:03:36.124149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:36.124149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.124451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:36.124451 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.124451 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:36.124451 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:36.124451 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.124451 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:36.124451 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.124451 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:36.125206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:36.125571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:36.125571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:36.125571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.125571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:36.125571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:36.125571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.125571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:36.125571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.125571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:36.125737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.125737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:36.125737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:36.125737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:36.125737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.125737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:36.125737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:36.125737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:03:36.125737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:36.125737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:36.125737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.125737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:36.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:36.126240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.126240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:36.126240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.126240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:36.126555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:36.127180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:36.127538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.127538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:36.127538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:36.127538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.127538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:36.127538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.127538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:36.133468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:36.133978 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:36.134148 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:36.134333 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:36.134785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:36.134979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.135415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:36.135415 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.135415 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:36.135415 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:36.135415 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.135415 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:36.135415 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.135415 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:36.135982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:36.136298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:36.136298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:36.136298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.136298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:36.136298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:36.136298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.136298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:36.136298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.136298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:36.136640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.136640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:36.136640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:36.136640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:36.136640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.136640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:36.136640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:36.136640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:36.136640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:36.136640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.136640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:36.136828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:36.137067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.137067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:36.137067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.137067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.137484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:36.137836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:36.138326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.138326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:36.138326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.138326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:36.149057 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:36.149486 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:36.149585 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:36.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:36.150183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:36.150438 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:36.150683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.153147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:36.153147 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.153147 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:36.153147 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:36.153147 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.153147 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:36.153147 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.153147 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:36.154168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.154168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:36.154168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:36.154168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:36.154168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.154168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:36.154168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:36.154168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.154168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:36.154168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:36.154168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:36.154168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.154462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:36.154679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.154679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:36.155368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:36.155692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:36.156103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:36.156103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:36.156103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:41.014564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:41.015385 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:41.015385 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:41.015631 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:41.061655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:41.061855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.062461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:41.062461 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.062461 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:41.062461 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:41.062461 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.062461 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:41.062461 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.062461 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:41.063512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.063512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:41.063512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:41.063512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:41.063512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.063512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:41.063512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:41.063512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.063512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:41.063512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:41.063512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:41.063512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.063747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:41.064014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.064014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:41.064584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:41.064931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.065357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:41.065357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.065357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:41.112223 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:41.112629 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:41.112723 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:41.114536 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:41.114713 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:41.114851 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:41.114973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:41.115968 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:41.115968 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:41.115968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.115968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:41.115968 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.115968 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:41.115968 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:41.115968 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.115968 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:41.115968 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.115968 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:41.116568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.116568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:41.116568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:41.116568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:41.116568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.116568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:41.116568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:41.116568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.116568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:41.116568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:41.116568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:41.116568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.116904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:41.117140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.117140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:41.117290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:41.117694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.118106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:41.118106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.118106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:41.123157 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:41.124653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:41.124653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:41.124653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:03:41.124653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:41.124653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.124841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:41.124841 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.124841 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:41.124841 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:41.124841 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.124841 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:41.124841 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.124841 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:41.125344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:41.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:41.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:41.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:41.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:41.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:41.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.125732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:41.125960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.125960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:41.125960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:41.125960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:41.125960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.125960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:41.125960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:41.125960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:03:41.125960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:41.125960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:41.125960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.125960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:41.126258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:41.126572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.126572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:41.126572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.126572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:41.126701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:41.126994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:41.127364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.127364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:41.127364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:41.127364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.127364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:41.127364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.127364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:41.134062 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:41.134490 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:41.134576 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:41.134688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:41.135010 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:41.135269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.135731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:41.135731 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.135731 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:41.135731 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:41.135731 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.135731 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:41.135731 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.135731 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:41.136522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:41.136781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:41.136781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:41.136781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.136781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:41.136781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:41.136781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.136781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:41.136781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.136781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:41.137558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.137558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:41.137558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:41.137558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:41.137558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.137558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:41.137558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:41.137558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:41.137558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:41.137558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.137558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:41.137948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:41.138339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.138339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:41.138339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.138339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.140217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:41.140580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:41.140868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.140868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:41.140868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.140868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:41.149592 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:41.149998 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:41.150138 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:41.150363 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:41.150689 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:41.150879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:41.151103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.151599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:41.151599 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.151599 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:41.151599 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:41.151599 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.151599 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:41.151599 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.151599 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:41.152305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.152305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:41.152305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:41.152305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:41.152305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.152305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:41.152305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:41.152305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.152305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:41.152305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:41.152305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:41.152305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.152553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:41.152767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.152767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:41.152997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:41.153356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:41.153703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:41.153703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:41.153703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:46.014958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:46.015426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:46.015480 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:46.016041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:46.043998 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:46.044304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.044981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:46.044981 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.044981 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:46.044981 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:46.044981 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.044981 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:46.044981 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.044981 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:46.045983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.045983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:46.045983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:46.045983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:46.045983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.045983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:46.045983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:46.045983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.045983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:46.045983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:46.045983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:46.045983 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.046440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:46.046690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.046690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:46.047137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:46.047509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.047926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:46.047926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.047926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:46.113558 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:46.113997 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:46.114251 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:46.115446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:46.115576 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:46.115763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:46.115927 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:46.116109 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:46.116382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:46.116419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.116978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:46.116978 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.116978 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:46.116978 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:46.116978 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.116978 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:46.116978 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.116978 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:46.117731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.117731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:46.117731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:46.117731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:46.117731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.117731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:46.117731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:46.117731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.117731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:46.117731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:46.117731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:46.117731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.117956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:46.118156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.118156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:46.118561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:46.118820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.119133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:46.119133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.119133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:46.123527 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:46.123958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:46.124184 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:46.124392 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:03:46.125601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:46.125820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.126419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:46.126419 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.126419 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:46.126419 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:46.126419 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.126419 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:46.126419 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.126419 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:46.127390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:46.127654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:46.127654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:46.127654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.127654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:46.127654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:46.127654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.127654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:46.127654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.127654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:46.128346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.128346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:46.128346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:46.128346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:46.128346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.128346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:46.128346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:46.128346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:03:46.128346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:46.128346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:46.128346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.128346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:46.128576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:46.128782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.128782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:46.128782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.128782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:46.129523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:46.130072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.130072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:46.130072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:46.130072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.130072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:46.130072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.130072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:46.134511 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:46.134881 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:46.135052 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:46.135241 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:46.135635 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:46.135948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.136461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:46.136461 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.136461 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:46.136461 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:46.136461 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.136461 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:46.136461 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.136461 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:46.137300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:46.137571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:46.137571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:46.137571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.137571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:46.137571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:46.137571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.137571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:46.137571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.137571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:46.138292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.138292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:46.138292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:46.138292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:46.138292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.138292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:46.138292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:46.138292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:46.138292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:46.138292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.138292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:46.138495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:46.138697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.138697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:46.138697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.138697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.139489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:46.140293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:46.140577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.140577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:46.140577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.140577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:46.150012 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:46.150293 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:46.150370 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:46.150569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:46.151208 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:46.151494 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:46.151767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.152448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:46.152448 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.152448 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:46.152448 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:46.152448 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.152448 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:46.152448 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.152448 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:46.153322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.153322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:46.153322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:46.153322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:46.153322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.153322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:46.153322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:46.153322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.153322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:46.153322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:46.153322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:46.153322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.153717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:46.154063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.154063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:46.154543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:46.155606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:46.156788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:46.156788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:46.156788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:51.015441 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:51.017574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:51.017574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:51.017574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:51.056076 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:51.056320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.056945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:51.056945 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.056945 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:51.056945 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:51.056945 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.056945 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:51.056945 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.056945 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:51.058354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.058354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:51.058354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:51.058354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:51.058354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.058354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:51.058354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:51.058354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.058354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:51.058354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:51.058354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:51.058354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.058641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:51.060515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:51.060819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.061244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:51.061244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:51.061244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.061244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:51.061244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.061244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:51.114100 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:51.114409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:51.114491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:51.115354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:51.115481 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:51.115583 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:51.115673 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:51.115888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:51.116050 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:51.116176 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:51.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.116757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:51.116757 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.116757 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:51.116757 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:51.116757 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.116757 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:51.116757 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.116757 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:51.117583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.117583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:51.117583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:51.117583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:51.117583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.117583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:51.117583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:51.117583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.117583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:51.117583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:51.117583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:51.117583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.117918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:51.118343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.118343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:51.119138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:51.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.120069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:51.120069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.120069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:51.123968 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:51.124292 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:51.124385 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:51.124470 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:03:51.124739 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:51.124949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.125333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:51.125333 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.125333 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:51.125333 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:51.125333 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.125333 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:51.125333 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.125333 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:51.125903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:51.126201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:51.126201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:51.126201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.126201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:51.126201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:51.126201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.126201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:51.126201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.126201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:51.126961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.126961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:51.126961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:51.126961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:51.126961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.126961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:51.126961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:51.126961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:03:51.126961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:51.126961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:51.126961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.126961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:51.127237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:51.127483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.127483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:51.127483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.127483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:51.127706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:51.127979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:51.128304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.128304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:51.128304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:51.128304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.128304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:51.128304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.128304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:51.134967 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:51.135245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:51.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:51.135443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:51.135687 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:51.135923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.136312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:51.136312 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.136312 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:51.136312 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:51.136312 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.136312 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:51.136312 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.136312 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:51.136870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:51.137240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:51.137240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:51.137240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.137240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:51.137240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:51.137240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.137240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:51.137240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.137240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:51.137787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.137787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:51.137787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:51.137787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:51.137787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.137787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:51.137787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:51.137787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:51.137787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:51.137787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.137787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:51.138049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:51.138307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.138307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:51.138307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.138307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.138787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:51.139112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:51.139487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.139487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:51.139487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.139487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:51.150988 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:51.151296 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:51.151382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:51.151511 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:51.151797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:51.152052 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:51.152287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.152684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:51.152684 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.152684 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:51.152684 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:51.152684 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.152684 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:51.152684 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.152684 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:51.153465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.153465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:51.153465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:51.153465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:51.153465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.153465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:51.153465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:51.153465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.153465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:51.153465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:51.153465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:51.153465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.153730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:51.153936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.153936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:51.155350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:51.155637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:51.155970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:51.155970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:51.155970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:56.016529 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:56.017657 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:56.017756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:56.018462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:56.050291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:56.050442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.050886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:56.050886 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.050886 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:56.050886 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:56.050886 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.050886 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:56.050886 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.050886 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:56.051526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.051526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:56.051526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:56.051526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:56.051526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.051526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:56.051526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:56.051526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.051526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:56.051526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:56.051526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:56.051526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.051923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:56.052299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:56.052697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.053173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:56.053173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:56.053173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.053173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:56.053173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.053173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:56.114610 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:56.115181 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:56.115328 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:56.116332 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:56.116506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:56.116660 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:56.116806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:56.116946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:56.117089 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:56.117254 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:56.117386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.117923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:56.117923 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.117923 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:56.117923 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:56.117923 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.117923 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:56.117923 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.117923 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:56.118869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.118869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:56.118869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:56.118869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:56.118869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.118869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:56.118869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:56.118869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.118869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:56.118869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:56.118869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:56.118869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.119290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:56.119619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.119619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:56.120005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:56.120540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.121071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:56.121071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.121071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:56.124439 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:56.124638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:56.124759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:56.124894 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:03:56.125298 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:56.125493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.126307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:56.126307 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.126307 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:56.126307 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:56.126307 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.126307 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:56.126307 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.126307 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:56.126791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:56.127264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:56.127264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:56.127264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.127264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:56.127264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:56.127264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.127264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:56.127264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.127264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:56.127684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.127684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:56.127684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:56.127684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:56.127684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.127684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:56.127684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:56.127684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:03:56.127684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:56.127684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:56.127684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.127684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:56.128068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:56.128469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.128469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:56.128469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.128469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:56.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:56.129303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:56.129794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.129794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:56.129794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:56.129794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.129794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:56.129794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.129794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:56.135476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:56.135686 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:56.135802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:56.135949 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:56.136291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:56.136537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.137086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:56.137086 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.137086 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:56.137086 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:56.137086 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.137086 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:56.137086 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.137086 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:03:56.137849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:03:56.138331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:56.138331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:03:56.138331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.138331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:03:56.138331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:03:56.138331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.138331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:03:56.138331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.138331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:03:56.138801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.138801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:56.138801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:56.138801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:56.138801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.138801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:56.138801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:56.138801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:56.138801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:56.138801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.138801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:56.139132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:56.139517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.139517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:56.139517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.139517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.140003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:56.140590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:56.141130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.141130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:56.141130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.141130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:03:56.151591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:03:56.151835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:03:56.151954 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:03:56.152147 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:03:56.152491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:56.152738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:03:56.152999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.153594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:03:56.153594 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.153594 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:03:56.153594 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:03:56.153594 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.153594 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:03:56.153594 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.153594 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:03:56.154555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.154555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:03:56.154555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:56.154555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:03:56.154555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.154555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:56.154555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:56.154555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.154555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:56.154555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:56.154555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:03:56.154555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.154964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:03:56.155334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.155334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:03:56.155710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:03:56.156176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:03:56.156710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:03:56.156710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:03:56.156710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:01.019478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:01.020867 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:01.020976 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:01.021637 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:01.050994 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:01.051157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.051565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:01.051565 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.051565 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:01.051565 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:01.051565 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.051565 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:01.051565 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.051565 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:01.052220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.052220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:01.052220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:01.052220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:01.052220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.052220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:01.052220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:01.052220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.052220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:01.052220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:01.052220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:01.052220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.052582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:01.052890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.052890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:01.052960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:01.053540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.053963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:01.053963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.053963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:01.115510 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:01.115890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:01.115961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:01.116744 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:01.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:01.117032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:01.117203 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:01.117461 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:01.117461 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:01.117538 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:01.117592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.117997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:01.117997 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.117997 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:01.117997 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:01.117997 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.117997 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:01.117997 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.117997 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:01.120032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.120032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:01.120032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:01.120032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:01.120032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.120032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:01.120032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:01.120032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.120032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:01.120032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:01.120032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:01.120032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.120440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:01.120810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:01.121274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.121800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:01.121800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:01.121800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.121800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:01.121800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.121800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:01.125049 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:01.125417 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:01.125524 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:01.125654 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:04:01.125938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:01.126162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.126680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:01.126680 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.126680 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:01.126680 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:01.126680 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.126680 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:01.126680 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.126680 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:01.128229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:01.128714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:01.129150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:01.129564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:01.129564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:01.129564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:01.129564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:01.129564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:01.129564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.129564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:01.129564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:01.129564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.129564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:01.129564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.129564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:01.129924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:01.130404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:01.130699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.130699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:01.130699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:01.130699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.130699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:01.130699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.130699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:01.136380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:01.136380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:01.136380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:01.136380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:01.136627 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:01.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.137238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:01.137238 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.137238 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:01.137238 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:01.137238 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.137238 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:01.137238 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.137238 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:01.137728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:01.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:01.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:01.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:01.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:01.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:01.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.138149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:01.138423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.138423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:01.138423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:01.138423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:01.138423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.138423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:01.138423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:01.138423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:01.138423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:01.138423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.138423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:01.138773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:01.139064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.139064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:01.139064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.139064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.139140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:01.140179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:01.140852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.140852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:01.140852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.140852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:01.152045 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:01.152277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:01.152350 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:01.152518 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:01.152765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:01.152941 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:01.153167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.153577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:01.153577 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.153577 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:01.153577 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:01.153577 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.153577 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:01.153577 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.153577 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:01.154155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.154155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:01.154155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:01.154155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:01.154155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.154155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:01.154155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:01.154155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.154155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:01.154155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:01.154155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:01.154155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.154567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:01.155203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:01.155633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:01.156205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:01.156205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:01.156205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:01.156205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:01.156205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:01.156205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:06.022309 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:06.022309 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:06.022309 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:06.022309 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:06.066158 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:06.066405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.066882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:06.066882 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.066882 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:06.066882 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:06.066882 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.066882 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:06.066882 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.066882 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:06.067669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.067669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:06.067669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:06.067669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:06.067669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.067669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:06.067669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:06.067669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.067669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:06.067669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:06.067669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:06.067669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.067945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:06.068173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.068173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:06.068793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:06.069092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.069431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:06.069431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.069431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:06.116011 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:06.116470 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:06.116651 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:06.117633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:06.117895 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:06.118125 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:06.118409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:06.118609 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:06.118797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:06.118968 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:06.119130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.119661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:06.119661 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.119661 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:06.119661 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:06.119661 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.119661 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:06.119661 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.119661 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:06.120544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.120544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:06.120544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:06.120544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:06.120544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.120544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:06.120544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:06.120544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.120544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:06.120544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:06.120544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:06.120544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.120842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:06.121122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.121122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:06.121960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:06.122370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.122748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:06.122748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.122748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:06.125482 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:06.125771 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:06.125910 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:06.126016 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:04:06.126384 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:06.126629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.127035 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:06.127035 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.127035 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:06.127035 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:06.127035 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.127035 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:06.127035 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.127035 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:06.127743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:06.128084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:06.128084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:06.128084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.128084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:06.128084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:06.128084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.128084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:06.128084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.128084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:06.128998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.128998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:06.128998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:06.128998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:06.128998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.128998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:06.128998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:06.128998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:04:06.128998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:06.128998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:06.128998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.128998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:06.129356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:06.129657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.129657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:06.129657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.129657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:06.129827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:06.130234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:06.130625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.130625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:06.130625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:06.130625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.130625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:06.130625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.130625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:06.136113 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:06.136347 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:06.136454 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:06.136596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:06.136963 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:06.137249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.137726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:06.137726 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.137726 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:06.137726 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:06.137726 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.137726 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:06.137726 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.137726 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:06.138678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:06.139104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:06.139104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:06.139104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.139104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:06.139104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:06.139104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.139104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:06.139104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.139104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:06.139907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.139907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:06.139907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:06.139907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:06.139907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.139907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:06.139907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:06.139907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:06.139907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:06.139907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.139907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:06.140245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:06.140571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.140571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:06.140571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.140571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.141331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:06.141626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:06.141930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.141930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:06.141930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.141930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:06.152340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:06.152720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:06.152813 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:06.152977 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:06.153375 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:06.153677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:06.153934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.154463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:06.154463 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.154463 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:06.154463 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:06.154463 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.154463 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:06.154463 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.154463 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:06.155301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.155301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:06.155301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:06.155301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:06.155301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.155301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:06.155301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:06.155301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.155301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:06.155301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:06.155301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:06.155301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.155612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:06.155899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.155899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:06.156630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:06.156968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:06.157477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:06.157477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:06.157477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:11.023256 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:11.024277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:11.024446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:11.025384 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:11.074721 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:11.074968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.075721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:11.075721 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.075721 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:11.075721 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:11.075721 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.075721 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:11.075721 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.075721 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:11.076899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.076899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:11.076899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:11.076899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:11.076899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.076899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:11.076899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:11.076899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.076899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:11.076899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:11.076899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:11.076899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.077287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:11.077708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.077708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:11.078208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:11.078722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.079164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:11.079164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.079164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:11.116519 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:11.116885 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:11.117121 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:11.118000 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:11.118284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:11.118416 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:11.118544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:11.118676 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:11.118785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:11.118882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:11.118981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:11.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:11.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:11.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:11.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.119612 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:11.120529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.120529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:11.120529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:11.120529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:11.120529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.120529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:11.120529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:11.120529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.120529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:11.120529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:11.120529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:11.120529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.120765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:11.120966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.120966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:11.121468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:11.121760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.122071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:11.122071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.122071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:11.125832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:11.126200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:11.126340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:11.126451 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:04:11.126756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:11.126919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.127392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:11.127392 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.127392 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:11.127392 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:11.127392 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.127392 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:11.127392 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.127392 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:11.128146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:11.128687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:11.128687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:11.128687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.128687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:11.128687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:11.128687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.128687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:11.128687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.128687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:11.129081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.129081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:11.129081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:11.129081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:11.129081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.129081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:11.129081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:11.129081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:04:11.129081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:11.129081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:11.129081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.129081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:11.129507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:11.129932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.129932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:11.129932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.129932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:11.130251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:11.130861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:11.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:11.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:11.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:11.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:11.136591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:11.136972 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:11.137089 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:11.137280 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:11.137885 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:11.138183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.138828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:11.138828 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.138828 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:11.138828 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:11.138828 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.138828 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:11.138828 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.138828 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:11.139682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:11.140016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:11.140016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:11.140016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.140016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:11.140016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:11.140016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.140016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:11.140016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.140016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:11.140599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.140599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:11.140599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:11.140599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:11.140599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.140599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:11.140599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:11.140599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:11.140599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:11.140599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.140599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:11.140854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:11.141178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.141178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:11.141178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.141178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.141661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:11.142177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:11.142607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.142607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:11.142607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.142607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:11.153135 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:11.153365 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:11.153583 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:11.153583 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:11.154277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:11.154568 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:11.154875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.155503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:11.155503 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.155503 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:11.155503 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:11.155503 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.155503 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:11.155503 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.155503 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:11.156229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.156229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:11.156229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:11.156229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:11.156229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.156229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:11.156229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:11.156229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.156229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:11.156229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:11.156229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:11.156229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.156658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:11.156974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.156974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:11.157239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:11.157692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:11.158099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:11.158099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:11.158099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:16.024255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:16.024684 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:16.024750 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:16.025383 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:16.055922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:16.055922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.056071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:16.056071 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.056071 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:16.056071 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:16.056071 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.056071 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:16.056071 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.056071 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:16.056755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.056755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:16.056755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:16.056755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:16.056755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.056755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:16.056755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:16.056755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.056755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:16.056755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:16.056755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:16.056755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.057298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:16.057773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:16.058277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.058585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:16.058585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:16.058585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.058585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:16.058585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.058585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:16.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:16.117495 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:16.117624 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:16.119633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:16.119633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:16.119633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:16.119633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:16.119633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:16.119890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:16.119890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:16.119890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.119890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:16.119890 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.119890 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:16.119890 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:16.119890 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.119890 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:16.119890 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.119890 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:16.120727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.120727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:16.120727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:16.120727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:16.120727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.120727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:16.120727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:16.120727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.120727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:16.120727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:16.120727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:16.120727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.121269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:16.121746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.121746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:16.121832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:16.122584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.123581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:16.123581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.123581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:16.126140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:16.126487 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:16.126611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:16.126745 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:04:16.127681 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:16.127681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.127681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:16.127681 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.127681 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:16.127681 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:16.127681 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.127681 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:16.127681 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.127681 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:16.128399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:16.128794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:16.128794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:16.128794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.128794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:16.128794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:16.128794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.128794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:16.128794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.128794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:16.129416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.129416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:16.129416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:16.129416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:16.129416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.129416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:16.129416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:16.129416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:04:16.129416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:16.129416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:16.129416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.129416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:16.130118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:16.130636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:16.130933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:16.131270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:16.131270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:16.131270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:16.131270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:16.131270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:16.131270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.131270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:16.131270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:16.131270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.131270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:16.131270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.131270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:16.136900 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:16.137232 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:16.137320 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:16.137422 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:16.138440 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:16.138440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:16.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:16.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:16.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:16.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:16.139083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:16.139504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:16.139504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:16.139504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.139504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:16.139504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:16.139504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.139504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:16.139504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.139504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:16.139754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.139754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:16.139754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:16.139754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:16.139754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.139754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:16.139754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:16.139754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:16.139754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:16.139754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.139754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:16.140079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:16.140473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:16.140839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:16.141362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:16.141362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:16.141362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:16.141362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:16.141362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.141362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:16.141362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:16.141362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.141362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:16.141362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.141362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:16.153255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:16.153561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:16.153653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:16.153786 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:16.154721 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:16.154721 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:16.154721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.154808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:16.154808 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.154808 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:16.154808 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:16.154808 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.154808 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:16.154808 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.154808 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:16.155393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.155393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:16.155393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:16.155393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:16.155393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.155393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:16.155393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:16.155393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.155393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:16.155393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:16.155393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:16.155393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.155607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:16.155804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.155804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:16.156068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:16.156632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:16.157208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:16.157208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:16.157208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:21.026581 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:21.026581 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:21.026581 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:21.026581 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:21.027646 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=239 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:04:21.063989 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.064206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:21.065870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:21.066483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.066715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:21.066715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:21.066715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.066715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:21.066715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.066715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:21.067674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:21.068032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.068440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:21.068440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.068440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:21.117514 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:21.117931 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:21.118036 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:21.120125 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.120330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.120443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.120575 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.120706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.120805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.120904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.120995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.121351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.122078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:21.122078 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.122078 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:21.122078 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:21.122078 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.122078 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:21.122078 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.122078 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:21.124968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.124968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:21.124968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:21.124968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:21.124968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.124968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:21.124968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:21.124968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.124968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:21.124968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:21.124968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:21.124968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.125715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:21.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:21.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.127013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:21.127013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:21.127013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.127013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:21.127013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.127013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:21.127383 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:21.127865 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:21.128062 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:21.128335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:04:21.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.130371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.130832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:21.130832 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.130832 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:21.130832 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:21.130832 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.130832 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:21.130832 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.130832 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:21.132988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:21.133495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:21.133784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:21.134013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:21.134013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:21.134013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:21.134013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:21.134013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:21.134013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.134013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:21.134013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:21.134013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.134013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:21.134013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.134013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:21.134220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:21.134630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:21.135085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.135085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:21.135085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:21.135085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.135085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:21.135085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.135085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:21.137351 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:21.138144 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:21.138144 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:21.138144 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:21.138473 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.138674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.139068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:21.139068 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.139068 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:21.139068 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:21.139068 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.139068 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:21.139068 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.139068 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:21.139825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:21.140145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:21.140145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:21.140145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.140145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:21.140145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:21.140145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.140145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:21.140145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.140145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:21.141831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.141831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:21.141831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:21.141831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:21.141831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.141831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:21.141831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:21.141831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:21.141831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:21.141831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.141831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:21.142119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:21.142354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:21.142608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:21.142941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:21.142941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:21.142941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:21.142941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:21.142941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.142941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:21.142941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:21.142941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.142941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:21.142941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.142941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:21.153676 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:21.154426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:21.154426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:21.154426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:21.155752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.155948 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:21.156153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.156697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:21.156697 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.156697 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:21.156697 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:21.156697 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.156697 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:21.156697 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.156697 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:21.157612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.157612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:21.157612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:21.157612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:21.157612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.157612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:21.157612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:21.157612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.157612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:21.157612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:21.157612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:21.157612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.157882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:21.159284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:21.159630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:21.159951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:21.159951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:21.159951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:21.159951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:21.159951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:21.159951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:26.027294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:26.027294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:26.027294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:26.027294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:26.081780 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.082038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.082844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:26.082844 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.082844 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:26.082844 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:26.082844 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.082844 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:26.082844 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.082844 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:26.083097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.083097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:26.083097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:26.083097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:26.083097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.083097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:26.083097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:26.083097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.083097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:26.083097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:26.083097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:26.083097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.083367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:26.083585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.083585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:26.083897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:26.084250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.084578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:26.084578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.084578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:26.117978 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:26.119953 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:26.120090 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:26.120883 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.121043 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.121178 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.121286 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.121393 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.121508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.121603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6429e3d8-2ead-48ae-a6c9-e35a93c0c30a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.121697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.121780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.122214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:26.122214 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.122214 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:26.122214 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:26.122214 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.122214 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:26.122214 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.122214 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:26.123033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.123033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:26.123033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:26.123033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:26.123033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.123033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:26.123033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:26.123033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.123033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:26.123033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:26.123033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:26.123033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.125000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:26.125817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:26.126454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.127679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:26.127679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:26.127679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.127679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:26.127679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.127679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:26.130120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:26.130120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:26.130120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:26.130120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:04:26.131401 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.131401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.131401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:26.131401 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.131401 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:26.131401 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:26.131401 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.131401 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:26.131401 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.131401 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:26.132013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:26.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:26.134284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:26.135163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:26.135163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:26.135163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:26.135163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:26.135163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:26.135163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.135163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:26.135163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:26.135163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.135163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:26.135163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.135163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:26.136040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:26.137301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:26.138157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.138157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:26.138157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:26.138157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.138157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:26.138157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.138157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:26.138623 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:26.143117 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:26.145741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:26.150410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:26.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.151741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.155670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:26.156692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:26.157120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.157961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:26.159442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:26.164728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:26.165286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:26.165286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.165286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:26.165286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:26.165286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.165286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:26.165286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.165286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:26.165286 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:26.165286 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:26.165286 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:26.165606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:26.165606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.165606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:26.165606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.165606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:26.165606 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.165606 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:26.165606 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:26.165606 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.165606 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:26.165606 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.165606 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:26.165941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.165941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:26.165941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:26.165941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:26.165941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.165941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:26.165941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:26.165941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.165941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:26.165941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:26.165941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:26.165941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.166298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:26.166707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:26.167119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:26.168567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:26.168567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:26.168567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:26.168567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:26.168567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:26.168567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:31.026469 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:31.028972 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:31.028972 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:31.028972 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:31.081435 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.081435 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:31.083889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:31.084476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.084756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:31.085076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:31.085471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:31.085471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:31.085471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.085471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:31.085471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:31.085471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.085471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:31.085471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.085471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:31.120128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:31.121472 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:31.121472 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:31.121675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.121865 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.122074 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.122831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.123009 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.123248 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.123420 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.123634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:31.126242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:31.126721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.127162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:31.127557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:31.128070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:31.128070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:31.128070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:31.128070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.128070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:31.128070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:31.128070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.128070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:31.128070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.128070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:31.128463 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:31.128936 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:31.129148 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:31.129361 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:04:31.129772 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.129957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.131313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:31.131313 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.131313 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:31.131313 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:31.131313 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.131313 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:31.131313 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.131313 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:31.132803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:31.133512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:31.133512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:31.133512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.133512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:31.133512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:31.133512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.133512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:31.133512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.133512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:31.135033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.135033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:31.135033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:31.135033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:31.135033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.135033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:31.135033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:31.135033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:04:31.135033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:31.135033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:31.135033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.135033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:31.135429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:31.135777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.135777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:31.135777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.135777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:31.138117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:31.139407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:31.140556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.140556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:31.140556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:31.140556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.140556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:31.140556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.140556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:31.143204 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:31.144179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:31.144179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:31.144179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:31.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.144974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.147262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:31.147262 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.147262 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:31.147262 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:31.147262 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.147262 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:31.147262 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.147262 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:31.148715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:31.149324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:31.149324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:31.149324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.149324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:31.149324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:31.149324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.149324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:31.149324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.149324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:31.150741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.150741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:31.150741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:31.150741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:31.150741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.150741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:31.150741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:31.150741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:31.150741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:31.150741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.150741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:31.151066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:31.151428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.151428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:31.151428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.151428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.152836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:31.153473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:31.153931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.153931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:31.153931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.153931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:31.158705 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:31.159140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:31.159140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:31.159301 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:31.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.161609 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:31.161860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.162481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:31.162481 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.162481 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:31.162481 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:31.162481 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.162481 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:31.162481 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.162481 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:31.164403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.164403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:31.164403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:31.164403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:31.164403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.164403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:31.164403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:31.164403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.164403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:31.164403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:31.164403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:31.164403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.164754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:31.165238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.165238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:31.166167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:31.166656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:31.167170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:31.167170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:31.167170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:36.027430 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:36.027882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:36.027882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:36.028554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:36.065246 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.065246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.065514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:36.065514 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.065514 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:36.065514 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:36.065514 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.065514 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:36.065514 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.065514 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:36.066630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.066630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:36.066630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:36.066630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:36.066630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.066630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:36.066630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:36.066630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.066630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:36.066630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:36.066630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:36.066630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.067209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:36.070587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:36.070895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.071240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:36.071240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:36.071240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.071240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:36.071240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.071240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:36.120737 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:36.122481 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:36.122481 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:36.124396 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.124396 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.124396 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.124396 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.124396 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.124585 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.124585 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.124585 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.124585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.124779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:36.124779 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.124779 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:36.124779 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:36.124779 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.124779 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:36.124779 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.124779 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:36.125667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.125667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:36.125667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:36.125667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:36.125667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.125667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:36.125667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:36.125667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.125667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:36.125667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:36.125667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:36.125667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.126078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:36.126388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.126388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:36.126499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:36.126753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.127062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:36.127062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.127062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:36.129083 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:36.129413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:36.129532 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:36.129660 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:04:36.130485 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.130485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.130639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:36.130639 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.130639 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:36.130639 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:36.130639 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.130639 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:36.130639 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.130639 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:36.131341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:36.131912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:36.132791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:36.133261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:36.133261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:36.133261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:36.133261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:36.133261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:36.133261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.133261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:36.133261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:36.133261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.133261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:36.133261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.133261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:36.133479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:36.133775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:36.134070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.134070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:36.134070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:36.134070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.134070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:36.134070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.134070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:36.143886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:36.144330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:36.144547 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:36.144680 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:36.145650 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.145650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.145814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:36.145814 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.145814 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:36.145814 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:36.145814 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.145814 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:36.145814 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.145814 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:36.146567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:36.146846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:36.146846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:36.146846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.146846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:36.146846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:36.146846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.146846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:36.146846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.146846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:36.147536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.147536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:36.147536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:36.147536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:36.147536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.147536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:36.147536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:36.147536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:36.147536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:36.147536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.147536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:36.147784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:36.148023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.148023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:36.148023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.148023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.148728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:36.149401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:36.149786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.149786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:36.149786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.149786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:36.159037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:36.160339 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:36.160488 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:36.160699 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:36.161952 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.161952 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:36.161952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.162563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:36.162563 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.162563 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:36.162563 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:36.162563 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.162563 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:36.162563 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.162563 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:36.163667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.163667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:36.163667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:36.163667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:36.163667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.163667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:36.163667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:36.163667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.163667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:36.163667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:36.163667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:36.163667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.163886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:36.164108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.164108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:36.164826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:36.165319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:36.165909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:36.165909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:36.165909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:41.027800 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:41.029155 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:41.029155 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:41.029155 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:41.062552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.062803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.063385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:41.063385 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.063385 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:41.063385 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:41.063385 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.063385 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:41.063385 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.063385 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:41.064067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.064067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:41.064067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:41.064067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:41.064067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.064067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:41.064067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:41.064067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.064067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:41.064067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:41.064067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:41.064067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.064357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:41.064589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.064589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:41.064863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:41.065165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.065521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:41.065521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.065521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:41.121595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:41.121940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:41.122068 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:41.124256 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.124436 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.124544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.124644 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.124862 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.124999 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.125102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.125225 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.125351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:41.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:41.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:41.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:41.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.125962 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:41.126835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.126835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:41.126835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:41.126835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:41.126835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.126835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:41.126835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:41.126835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.126835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:41.126835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:41.126835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:41.126835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.127102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:41.127352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.127352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:41.129101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:41.129457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.129764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:41.129764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.129764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:41.129764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:41.129764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:41.129764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:41.129764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:04:41.129961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.130164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.130578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:41.130578 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.130578 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:41.130578 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:41.130578 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.130578 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:41.130578 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.130578 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:41.131637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:41.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:41.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:41.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:41.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:41.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:41.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.131963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:41.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:41.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:41.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:41.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:41.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:41.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:04:41.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:41.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:41.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:41.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:41.132842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.132842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:41.132842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.132842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:41.133465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:41.133782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:41.134101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.134101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:41.134101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:41.134101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.134101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:41.134101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.134101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:41.144364 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:41.144684 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:41.144874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:41.144965 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:41.145486 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.145744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.146379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:41.146379 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.146379 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:41.146379 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:41.146379 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.146379 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:41.146379 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.146379 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:41.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:41.147649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:41.147649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:41.147649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.147649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:41.147649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:41.147649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.147649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:41.147649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.147649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:41.148400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.148400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:41.148400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:41.148400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:41.148400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.148400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:41.148400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:41.148400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:41.148400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:41.148400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.148400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:41.148640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:41.148880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.148880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:41.148880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.148880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.150506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:41.150865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:41.151295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.151295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:41.151295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.151295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:41.160326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:41.160634 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:41.160708 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:41.160918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:41.161526 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.161732 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:41.161933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.162444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:41.162444 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.162444 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:41.162444 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:41.162444 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.162444 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:41.162444 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.162444 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:41.163516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.163516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:41.163516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:41.163516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:41.163516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.163516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:41.163516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:41.163516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.163516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:41.163516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:41.163516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:41.163516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.163850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:41.164155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.164155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:41.165394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:41.165871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:41.166406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:41.166406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:41.166406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:46.028340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:46.029323 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:46.029501 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:46.030120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:46.082093 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.082279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.082743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:46.082743 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.082743 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:46.082743 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:46.082743 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.082743 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:46.082743 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.082743 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:46.083347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.083347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:46.083347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:46.083347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:46.083347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.083347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:46.083347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:46.083347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.083347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:46.083347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:46.083347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:46.083347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.083794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:46.084149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.084149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:46.086250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:46.086963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.087466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:46.087466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.087466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:46.121949 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:46.122869 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:46.122869 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:46.123112 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.123291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.123419 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.123536 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.123649 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.123814 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.123858 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.123954 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-65f33571-0494-499f-be92-d5ac93b22e88, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.124046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:46.126283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:46.126634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.126888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:46.127138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:46.127449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:46.127449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:46.127449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:46.127449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.127449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:46.127449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:46.127449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.127449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:46.127449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.127449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:46.129692 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:46.129916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:46.130105 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:46.130327 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:04:46.130699 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.130976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.133136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:46.133598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:46.133918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:46.134459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:46.134780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:46.135106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:46.135106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:46.135106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.135106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:46.135106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:46.135106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.135106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:46.135106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.135106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:46.144731 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:46.145319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:46.145319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:46.145319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:46.145485 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.145669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.148064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:46.148515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:46.148872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.149114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:46.149370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:46.149632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:46.149997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:46.149997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.149997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:46.149997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:46.149997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.149997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:46.149997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.149997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:46.161151 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:46.161151 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:46.161151 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:46.161151 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:46.161443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.161443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:46.161674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:46.164050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:46.165155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.166174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:46.167128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:46.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:46.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:46.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:46.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:46.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:46.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:46.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:46.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:46.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:46.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:51.029443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:51.032257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:51.032257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:51.032257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:51.084331 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:51.084548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:51.086674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:51.087432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.087700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:51.087968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:51.088408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:51.088408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:51.088408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.088408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:51.088408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:51.088408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.088408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:51.088408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.088408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:51.122413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:51.123543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:51.123543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:51.123898 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:51.124179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:51.124358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:51.124495 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:51.124652 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:51.124795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:51.124914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:51.125052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.125727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:51.125727 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.125727 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:51.125727 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:51.125727 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.125727 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:51.125727 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.125727 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:51.127636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.127636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:51.127636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:51.127636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:51.127636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.127636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:51.127636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:51.127636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.127636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:51.127636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:51.127636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:51.127636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.127903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:51.128133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:51.128398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.128730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:51.128730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:51.128730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.128730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:51.128730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.128730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:51.130039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:51.130333 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:51.130511 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:51.130676 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:04:51.131106 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:51.131294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.133688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:51.134122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:51.134404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.134631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:51.134847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:51.135095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:51.135407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:51.135407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:51.135407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.135407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:51.135407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:51.135407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.135407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:51.135407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.135407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:51.145147 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:51.145704 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:51.145704 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:51.145704 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:51.145916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:51.146176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.146619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:51.146619 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.146619 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:51.146619 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:51.146619 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.146619 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:51.146619 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.146619 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:51.147406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:51.147707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:51.147707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:51.147707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.147707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:51.147707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:51.147707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.147707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:51.147707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.147707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:51.149758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.149758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:51.149758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:51.149758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:51.149758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.149758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:51.149758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:51.149758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:51.149758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:51.149758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.149758 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:51.150082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:51.150322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:51.150617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:51.150929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:51.150929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:51.150929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:51.150929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:51.150929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.150929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:51.150929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:51.150929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.150929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:51.150929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.150929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:51.161064 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:51.161730 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:51.161730 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:51.161730 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:51.162147 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:51.162411 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:51.162632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:51.164464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:51.164792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.165020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:51.165298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:51.165665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:51.165665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:51.165665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:51.165665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:51.165665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:51.165665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:51.165665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:51.165665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:51.165665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:51.165665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:56.031133 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:56.031755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:56.031755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:56.032284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:56.077536 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:56.077536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.078667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:56.078667 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.078667 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:56.078667 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:56.078667 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.078667 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:56.078667 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.078667 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:56.079614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.079614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:56.079614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:56.079614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:56.079614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.079614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:56.079614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:56.079614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.079614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:56.079614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:56.079614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:56.079614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.080076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:56.080511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.080511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:56.081084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:56.081750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.082647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:56.082647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.082647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:56.124139 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:56.124697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:56.124697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:56.127651 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:56.127922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:56.128108 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:56.128262 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:56.128373 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:56.128489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:56.128573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:56.128738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.129383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:56.129383 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.129383 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:56.129383 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:56.129383 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.129383 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:56.129383 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.129383 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:56.131220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.131220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:56.131220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:56.131220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:56.131220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.131220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:56.131220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:56.131220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.131220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:56.131220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:56.131220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:56.131220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.131693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:56.132097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.132097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:56.132582 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:56.134020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:56.134407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.134766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:56.134766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.134766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:56.134766 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:56.134766 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:56.134766 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:04:56.134766 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:56.134766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.136283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:56.136283 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.136283 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:56.136283 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:56.136283 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.136283 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:56.136283 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.136283 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:56.137053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:56.137321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:56.137321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:56.137321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.137321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:56.137321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:56.137321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.137321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:56.137321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.137321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:56.140446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.140446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:56.140446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:56.140446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:56.140446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.140446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:56.140446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:56.140446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:04:56.140446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:56.140446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:56.140446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.140446 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:56.141100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:56.141690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:56.142101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:56.142630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:56.142630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:56.142630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:56.142630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:56.142630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:56.142630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.142630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:56.142630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:56.142630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.142630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:56.142630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.142630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:56.145630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:56.146524 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:56.146739 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:56.146974 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:56.147429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:56.147685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.148246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:56.148246 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.148246 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:56.148246 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:56.148246 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.148246 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:56.148246 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.148246 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:04:56.149768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:04:56.150042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:56.150042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:04:56.150042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.150042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:04:56.150042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:04:56.150042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.150042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:04:56.150042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.150042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:04:56.152183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.152183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:56.152183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:56.152183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:56.152183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.152183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:56.152183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:56.152183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:56.152183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:56.152183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.152183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:56.152450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:56.152664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.152664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:56.152664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.152664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.154158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:56.154616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:56.154945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.154945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:56.154945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.154945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:04:56.161562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:04:56.162178 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:04:56.162398 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:04:56.162906 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:04:56.163320 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:56.163577 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:04:56.163710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.164202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:04:56.164202 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.164202 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:04:56.164202 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:04:56.164202 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.164202 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:04:56.164202 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.164202 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:04:56.165025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.165025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:04:56.165025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:56.165025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:04:56.165025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.165025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:56.165025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:56.165025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.165025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:56.165025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:56.165025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:04:56.165025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.165347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:04:56.165602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.165602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:04:56.167503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:04:56.168178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:04:56.168570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:04:56.168570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:04:56.168570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:01.031633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:01.033695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:01.033695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:01.033695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:01.088720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.088940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:01.091360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:01.092044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.092409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:01.092753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:01.093302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:01.093302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:01.093302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.093302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:01.093302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:01.093302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.093302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:01.093302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.093302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:01.125899 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:01.128772 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:01.129694 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:01.129976 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.130419 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.130699 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.130878 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.131080 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.131175 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.131302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.131432 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.131531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:01.133382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:01.133734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.134008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:01.134298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:01.134657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:01.134657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:01.134657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:01.134657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.134657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:01.134657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:01.134657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.134657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:01.134657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.134657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:01.134657 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:01.135335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:01.135551 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:01.135752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:05:01.137539 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.138355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.138930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:01.138930 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.138930 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:01.138930 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:01.138930 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.138930 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:01.138930 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.138930 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:01.139990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:01.140710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:01.140710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:01.140710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.140710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:01.140710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:01.140710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.140710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:01.140710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.140710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:01.142114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.142114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:01.142114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:01.142114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:01.142114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.142114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:01.142114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:01.142114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:05:01.142114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:01.142114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:01.142114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.142114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:01.142420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:01.142628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.142628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:01.142628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.142628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:01.143955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:01.144727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:01.145145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.145145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:01.145145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:01.145145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.145145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:01.145145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.145145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:01.146582 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:01.147281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:01.147281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:01.147281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:01.147652 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.147964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.148478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:01.148478 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.148478 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:01.148478 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:01.148478 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.148478 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:01.148478 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.148478 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:01.150798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:01.151252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:01.151834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:01.152076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:01.152076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:01.152076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:01.152076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:01.152076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.152076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:01.152076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:01.152076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.152076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:01.152076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.152076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:01.152076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:01.152290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.152629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:01.162336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:01.162932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:01.162932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:01.162932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:01.163485 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.163814 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:01.166259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:01.166560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:01.166778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:01.167523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:01.167523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:01.167523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:01.167523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:01.167523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:01.167523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:01.167523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:01.167523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:01.167523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:01.167523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:01.167523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:01.167523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:06.032923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:06.033505 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:06.033562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:06.034367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:06.085727 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.085928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.086574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:06.086574 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.086574 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:06.086574 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:06.086574 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.086574 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:06.086574 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.086574 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:06.087371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.087371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:06.087371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:06.087371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:06.087371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.087371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:06.087371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:06.087371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.087371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:06.087371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:06.087371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:06.087371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.087739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:06.088079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:06.088402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.088777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:06.088777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:06.088777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.088777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:06.088777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.088777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:06.128100 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:06.128854 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:06.128981 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:06.129902 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.130066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.130149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.130998 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.130998 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.131423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.131423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.131423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.131528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.131883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:06.131883 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.131883 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:06.131883 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:06.131883 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.131883 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:06.131883 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.131883 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:06.132697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.132697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:06.132697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:06.132697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:06.132697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.132697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:06.132697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:06.132697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.132697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:06.132697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:06.132697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:06.132697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:06.133249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.133249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:06.133648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:06.134145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.134578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:06.134578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.134578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:06.135387 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:06.135640 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:06.135709 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:06.135799 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:05:06.136096 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.136280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.136642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:06.136642 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.136642 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:06.136642 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:06.136642 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.136642 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:06.136642 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.136642 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:06.137235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:06.137482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:06.137482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:06.137482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.137482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:06.137482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:06.137482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.137482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:06.137482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.137482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:06.137908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.137908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:06.137908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:06.137908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:06.137908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.137908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:06.137908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:06.137908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:05:06.137908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:06.137908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:06.137908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.137908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:06.138437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:06.138654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:06.138864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:06.139165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:06.139165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:06.139165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:06.139165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:06.139165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:06.139165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.139165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:06.139165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:06.139165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.139165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:06.139165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.139165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:06.146923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:06.147214 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:06.147312 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:06.147416 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:06.147706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.147943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.148387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:06.148387 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.148387 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:06.148387 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:06.148387 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.148387 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:06.148387 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.148387 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:06.148889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:06.149177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:06.149177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:06.149177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.149177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:06.149177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:06.149177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.149177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:06.149177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.149177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:06.149652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.149652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:06.149652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:06.149652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:06.149652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.149652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:06.149652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:06.149652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:06.149652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:06.149652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.149652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:06.149853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:06.150074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.150074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:06.150074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.150074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.150432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:06.152649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:06.153118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.153118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:06.153118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.153118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:06.162803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:06.163198 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:06.163309 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:06.163450 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:06.163781 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.164056 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:06.164390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.164901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:06.164901 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.164901 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:06.164901 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:06.164901 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.164901 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:06.164901 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.164901 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:06.165536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.165536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:06.165536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:06.165536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:06.165536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.165536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:06.165536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:06.165536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.165536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:06.165536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:06.165536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:06.165536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.166397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:06.167136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:06.167419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:06.167735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:06.167735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:06.167735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:06.167735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:06.167735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:06.167735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:11.032821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:11.033636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:11.033815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:11.034609 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:11.072387 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.072647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.073329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:11.073329 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.073329 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:11.073329 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:11.073329 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.073329 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:11.073329 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.073329 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:11.074137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.074137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:11.074137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:11.074137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:11.074137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.074137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:11.074137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:11.074137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.074137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:11.074137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:11.074137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:11.074137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.074409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:11.074684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.074684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:11.074840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:11.075143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.075584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:11.075584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.075584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:11.129309 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:11.129986 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:11.130175 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:11.132234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.132423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.132746 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.132746 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.132746 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.132895 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.132937 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.133057 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.133175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.133968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:11.133968 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.133968 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:11.133968 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:11.133968 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.133968 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:11.133968 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.133968 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:11.135958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.135958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:11.135958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:11.135958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:11.135958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.135958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:11.135958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:11.135958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.135958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:11.135958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:11.135958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:11.135958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.136357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:11.136705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.136705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:11.137376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:11.138128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:11.138605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.139105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:11.139105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.139105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:11.139859 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:11.139859 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:11.139859 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:05:11.140241 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.140421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.141207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:11.141207 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.141207 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:11.141207 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:11.141207 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.141207 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:11.141207 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.141207 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:11.141917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:11.142364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:11.142364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:11.142364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.142364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:11.142364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:11.142364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.142364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:11.142364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.142364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:11.143353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.143353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:11.143353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:11.143353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:11.143353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.143353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:11.143353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:11.143353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:05:11.143353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:11.143353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:11.143353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.143353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:11.143652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:11.143963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.143963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:11.143963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.143963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:11.144071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:11.144475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:11.144914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.144914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:11.144914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:11.144914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.144914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:11.144914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.144914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:11.147296 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:11.147666 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:11.148428 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:11.148428 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:11.148840 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.149075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.149571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:11.149571 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.149571 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:11.149571 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:11.149571 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.149571 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:11.149571 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.149571 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:11.150394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:11.150793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:11.150793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:11.150793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.150793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:11.150793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:11.150793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.150793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:11.150793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.150793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:11.152256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.152256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:11.152256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:11.152256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:11.152256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.152256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:11.152256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:11.152256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:11.152256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:11.152256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.152256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:11.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:11.153151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:11.155874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:11.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:11.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:11.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:11.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:11.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:11.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:11.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:11.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.158126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:11.163342 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:11.163617 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:11.164240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:11.164240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:11.164577 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.165003 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:11.165003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.166738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:11.166738 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.166738 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:11.166738 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:11.166738 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.166738 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:11.166738 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.166738 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:11.167608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.167608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:11.167608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:11.167608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:11.167608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.167608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:11.167608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:11.167608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.167608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:11.167608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:11.167608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:11.167608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.168418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:11.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:11.169851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:11.170673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:11.171119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:11.171119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:11.171119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:16.033740 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:16.034452 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:16.034452 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:16.035058 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:16.115071 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.115268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.115798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:16.115798 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.115798 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:16.115798 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:16.115798 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.115798 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:16.115798 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.115798 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:16.116805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.081 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.116805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:16.116805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:16.116805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:16.116805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.116805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:16.116805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:16.116805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.116805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:16.116805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:16.116805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:16.116805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.117017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:16.117252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.117252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:16.117856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:16.118126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.118431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:16.118431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.118431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:16.129461 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:16.129851 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:16.129973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:16.130949 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.131150 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.131291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.131422 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.131544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.131670 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.131814 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.131921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.132061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:16.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:16.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:16.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:16.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.132605 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:16.133518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.133518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:16.133518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:16.133518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:16.133518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.133518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:16.133518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:16.133518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.133518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:16.133518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:16.133518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:16.133518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.133809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:16.134273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.134273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:16.134793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:16.135372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.135855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:16.135855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.135855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:16.138489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:16.138838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:16.138956 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:16.139101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:05:16.139657 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.139907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.140498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:16.140498 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.140498 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:16.140498 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:16.140498 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.140498 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:16.140498 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.140498 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:16.141452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:16.141765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:16.141765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:16.141765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.141765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:16.141765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:16.141765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.141765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:16.141765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.141765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:16.142667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.142667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:16.142667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:16.142667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:16.142667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.142667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:16.142667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:16.142667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:05:16.142667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:16.142667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:16.142667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.142667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:16.143052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:16.143438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.143438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:16.143438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.143438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:16.144231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:16.145387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:16.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:16.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:16.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:16.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:16.147754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:16.148119 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:16.148253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:16.148369 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:16.148726 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.149011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.150104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:16.150104 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.150104 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:16.150104 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:16.150104 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.150104 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:16.150104 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.150104 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:16.150387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:16.150770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:16.150770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:16.150770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.150770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:16.150770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:16.150770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.150770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:16.150770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.150770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:16.151166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.151166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:16.151166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:16.151166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:16.151166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.151166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:16.151166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:16.151166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:16.151166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:16.151166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.151166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:16.151489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:16.151855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.151855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:16.151855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.151855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.152381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:16.152679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:16.153016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.153016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:16.153016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.153016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:16.163805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:16.164122 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:16.164287 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:16.164467 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:16.164873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.165212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:16.165558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.166227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:16.166227 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.166227 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:16.166227 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:16.166227 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.166227 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:16.166227 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.166227 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:16.167387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.167387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:16.167387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:16.167387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:16.167387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.167387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:16.167387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:16.167387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.167387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:16.167387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:16.167387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:16.167387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.167714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:16.168074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.168074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:16.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:16.169360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:16.169704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:16.169704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:16.169704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:21.034755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:21.034755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:21.035416 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:21.038213 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:21.040452 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=240 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:05:21.081409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.081611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.082414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:21.082414 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.082414 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:21.082414 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:21.082414 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.082414 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:21.082414 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.082414 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:21.083627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.083627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:21.083627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:21.083627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:21.083627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.083627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:21.083627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:21.083627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.083627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:21.083627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:21.083627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:21.083627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.083996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:21.084315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.084315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:21.085840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:21.086243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.086614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:21.086614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.086614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:21.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:21.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:21.131980 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:21.135101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.135101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.135101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.135101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.135297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.135297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.135402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.135516 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.135628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.136166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:21.136166 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.136166 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:21.136166 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:21.136166 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.136166 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:21.136166 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.136166 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:21.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:21.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:21.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:21.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:21.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:21.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:21.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:21.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:21.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.137070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:21.137330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.137330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:21.137592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:21.137953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.138285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:21.138285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.138285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:21.138793 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:21.139048 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:21.140886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:21.140886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:05:21.140886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.140886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.140886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:21.140886 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.140886 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:21.140886 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:21.140886 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.140886 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:21.140886 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.140886 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:21.141149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:21.141466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:21.141466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:21.141466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.141466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:21.141466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:21.141466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.141466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:21.141466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.141466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:21.141993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.141993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:21.141993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:21.141993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:21.141993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.141993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:21.141993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:21.141993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:05:21.141993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:21.141993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:21.141993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.141993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:21.142288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:21.142507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.142507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:21.142507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.142507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:21.143139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:21.143480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:21.143789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.143789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:21.143789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:21.143789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.143789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:21.143789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.143789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:21.148300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:21.149617 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:21.149689 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:21.149837 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:21.150654 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.150654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.150822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:21.150822 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.150822 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:21.150822 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:21.150822 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.150822 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:21.150822 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.150822 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:21.151804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:21.152106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:21.152106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:21.152106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.152106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:21.152106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:21.152106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.152106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:21.152106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.152106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:21.152604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.152604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:21.152604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:21.152604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:21.152604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.152604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:21.152604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:21.152604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:21.152604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:21.152604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.152604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:21.152840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:21.153117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.153117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:21.153117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.153117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.154863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:21.155299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:21.155715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.155715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:21.155715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.155715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:21.164248 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:21.164711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:21.164828 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:21.165016 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:21.165815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.166090 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:21.166392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.166917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:21.166917 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.166917 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:21.166917 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:21.166917 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.166917 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:21.166917 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.166917 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:21.167770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.167770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:21.167770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:21.167770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:21.167770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.167770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:21.167770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:21.167770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.167770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:21.167770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:21.167770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:21.167770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.168069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:21.168317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.168317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:21.168884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:21.169213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:21.169527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:21.169527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:21.169527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:26.034934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:26.035761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:26.036396 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:26.038204 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:26.080853 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.081155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.081792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:26.081792 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.081792 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:26.081792 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:26.081792 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.081792 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:26.081792 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.081792 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:26.082534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.082534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:26.082534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:26.082534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:26.082534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.082534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:26.082534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:26.082534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.082534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:26.082534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:26.082534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:26.082534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.082762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:26.082984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.082984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:26.083279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:26.083594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.083878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:26.083878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.083878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:26.131561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:26.131910 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:26.132027 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:26.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.133834 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.133995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.133995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.134199 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.134350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.134892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:26.134892 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.134892 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:26.134892 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:26.134892 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.134892 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:26.134892 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.134892 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:26.135619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.135619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:26.135619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:26.135619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:26.135619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.135619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:26.135619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:26.135619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.135619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:26.135619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:26.135619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:26.135619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.135920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:26.136227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.136227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:26.136287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:26.136552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.136864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:26.136864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.136864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:26.139521 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:26.139521 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:26.139521 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:26.139521 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:05:26.140744 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.140744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.140744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:26.140744 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.140744 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:26.140744 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:26.140744 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.140744 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:26.140744 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.140744 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:26.141215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:26.141507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:26.141507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:26.141507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.141507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:26.141507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:26.141507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.141507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:26.141507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.141507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:26.141911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.141911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:26.141911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:26.141911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:26.141911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.141911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:26.141911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:26.141911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:05:26.141911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:26.141911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:26.141911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.141911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:26.142197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:26.142458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.142458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:26.142458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.142458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:26.142599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:26.142866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:26.143273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.143273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:26.143273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:26.143273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.143273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:26.143273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.143273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:26.150091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:26.150091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:26.150091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:26.150091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:26.150263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.150510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.151016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:26.151016 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.151016 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:26.151016 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:26.151016 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.151016 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:26.151016 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.151016 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:26.151559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:26.151807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:26.151807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:26.151807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.151807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:26.151807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:26.151807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.151807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:26.151807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.151807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:26.152119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.152119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:26.152119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:26.152119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:26.152119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.152119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:26.152119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:26.152119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:26.152119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:26.152119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.152119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:26.152321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:26.152541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.152541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:26.152541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.152541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.152790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:26.153066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:26.153365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.153365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:26.153365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.153365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:26.164750 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:26.164992 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:26.165117 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:26.165272 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:26.165726 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.165935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:26.166204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.166644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:26.166644 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.166644 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:26.166644 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:26.166644 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.166644 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:26.166644 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.166644 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:26.167347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.167347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:26.167347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:26.167347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:26.167347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.167347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:26.167347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:26.167347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.167347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:26.167347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:26.167347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:26.167347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.167696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:26.168000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:26.168443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:26.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:26.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:26.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:26.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:26.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:26.168911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:31.035739 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:31.036228 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:31.036228 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:31.037115 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:31.074466 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.074466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.074563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:31.074563 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.074563 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:31.074563 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:31.074563 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.074563 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:31.074563 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.074563 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:31.075328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.075328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:31.075328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:31.075328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:31.075328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.075328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:31.075328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:31.075328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.075328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:31.075328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:31.075328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:31.075328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.075919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:31.076335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:31.076719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.077275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:31.077275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:31.077275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.077275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:31.077275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.077275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:31.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:31.132889 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:31.133009 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:31.136650 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.136958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.137205 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.137361 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.137487 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebeddbcd-1690-47d7-8f0d-e5ecc8345741, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.137601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.137714 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.137834 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.137937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.139438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:31.139438 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.139438 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:31.139438 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:31.139438 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.139438 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:31.139438 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.139438 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:31.140474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.140474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:31.140474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:31.140474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:31.140474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.140474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:31.140474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:31.140474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.140474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:31.140474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:31.140474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:31.140474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.140716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:31.140935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.140935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:31.141282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:31.141538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.141827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:31.141827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.141827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:31.141827 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:31.141827 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:31.141923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:31.142022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:05:31.142381 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.142533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.142902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:31.142902 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.142902 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:31.142902 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:31.142902 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.142902 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:31.142902 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.142902 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:31.143455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:31.143740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:31.143740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:31.143740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.143740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:31.143740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:31.143740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.143740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:31.143740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.143740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:31.144105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.144105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:31.144105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:31.144105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:31.144105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.144105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:31.144105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:31.144105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:05:31.144105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:31.144105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:31.144105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.144105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:31.144350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:31.144571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.144571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:31.144571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.144571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:31.144887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:31.145181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:31.145496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.145496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:31.145496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:31.145496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.145496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:31.145496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.145496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:31.149761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:31.150102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:31.150242 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:31.150358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:31.150700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.150899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.151358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:31.151358 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.151358 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:31.151358 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:31.151358 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.151358 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:31.151358 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.151358 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:31.151887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:31.152234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:31.152234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:31.152234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.152234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:31.152234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:31.152234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.152234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:31.152234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.152234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:31.152561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.152561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:31.152561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:31.152561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:31.152561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.152561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:31.152561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:31.152561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:31.152561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:31.152561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.152561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:31.152754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:31.152963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.152963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:31.152963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.152963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.153346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:31.153627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:31.153942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.153942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:31.153942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.153942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:31.165217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:31.165498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:31.165616 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:31.165767 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:31.166049 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.166399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:31.166459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.166907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:31.166907 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.166907 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:31.166907 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:31.166907 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.166907 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:31.166907 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.166907 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:31.167883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.167883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:31.167883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:31.167883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:31.167883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.167883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:31.167883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:31.167883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.167883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:31.167883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:31.167883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:31.167883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.168356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:31.168748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:31.169235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:31.169811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:31.169811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:31.169811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:31.169811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:31.169811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:31.169811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:36.036244 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:36.037082 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:36.037233 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:36.039988 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:36.079634 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:36.080302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.080302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:36.080302 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.080302 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:36.080302 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:36.080302 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.080302 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:36.080302 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.080302 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:36.080877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.080877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:36.080877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:36.080877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:36.080877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.080877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:36.080877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:36.080877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.080877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:36.080877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:36.080877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:36.080877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.081285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:36.081612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:36.082007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.082505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:36.082505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:36.082505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.082505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:36.082505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.082505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:36.132995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:36.133284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:36.133370 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:36.134345 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:36.134475 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:36.134946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:36.134946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:36.134946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:36.134946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:36.134946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:36.135237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.136066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:36.136066 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.136066 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:36.136066 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:36.136066 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.136066 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:36.136066 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.136066 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:36.136313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.136313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:36.136313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:36.136313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:36.136313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.136313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:36.136313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:36.136313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.136313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:36.136313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:36.136313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:36.136313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.136644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:36.136997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.136997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:36.137343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:36.138168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.138627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:36.138627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.138627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:36.141940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:36.142133 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:36.142254 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:36.142382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:05:36.143306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:36.143451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.143865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:36.143865 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.143865 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:36.143865 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:36.143865 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.143865 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:36.143865 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.143865 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:36.144490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:36.145376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:36.145763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:36.146141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:36.146141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:36.146141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:36.146141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:36.146141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:36.146141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.146141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:36.146141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:36.146141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.146141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:36.146141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.146141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:36.146525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:36.146963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:36.147515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.147515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:36.147515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:36.147515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.147515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:36.147515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.147515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:36.150050 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:36.150222 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:36.150303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:36.150405 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:36.150597 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:36.150755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.151080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:36.151080 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.151080 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:36.151080 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:36.151080 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.151080 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:36.151080 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.151080 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:36.151588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:36.152131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:36.152548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:36.153331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:36.153331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:36.153331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:36.153331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:36.153331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.153331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:36.153331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:36.153331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.153331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:36.153331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.153331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:36.153331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:36.153704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.154475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:36.165647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:36.165910 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:36.166051 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:36.166242 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:36.166547 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:36.166781 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:36.167024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.167732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:36.167732 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.167732 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:36.167732 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:36.167732 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.167732 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:36.167732 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.167732 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:36.168660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.168660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:36.168660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:36.168660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:36.168660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.168660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:36.168660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:36.168660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.168660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:36.168660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:36.168660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:36.168660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.169104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:36.169680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.169680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:36.169871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:36.170342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:36.170700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:36.170700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:36.170700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:41.037172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:41.037736 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:41.037944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:41.038871 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:41.075002 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.075257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.075641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:41.075641 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.075641 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:41.075641 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:41.075641 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.075641 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:41.075641 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.075641 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:41.076289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.076289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:41.076289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:41.076289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:41.076289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.076289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:41.076289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:41.076289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.076289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:41.076289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:41.076289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:41.076289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.076637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:41.077077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.077077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:41.077153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:41.077567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.078026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:41.078026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.078026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:41.133472 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:41.133971 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:41.134344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:41.134995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.135152 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.135320 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.135424 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.135510 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.135601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.135696 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.135783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.135957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.136429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:41.136429 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.136429 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:41.136429 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:41.136429 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.136429 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:41.136429 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.136429 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:41.137204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.137204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:41.137204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:41.137204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:41.137204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.137204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:41.137204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:41.137204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.137204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:41.137204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:41.137204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:41.137204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.137982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:41.138920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:41.139865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.140774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:41.140774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:41.140774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.140774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:41.140774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.140774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:41.142306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:41.142549 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:41.142648 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:41.142749 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:05:41.143154 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.143374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.143751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:41.143751 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.143751 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:41.143751 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:41.143751 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.143751 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:41.143751 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.143751 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:41.144312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:41.144742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:41.144742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:41.144742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.144742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:41.144742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:41.144742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.144742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:41.144742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.144742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:41.144964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.144964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:41.144964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:41.144964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:41.144964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.144964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:41.144964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:41.144964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:05:41.144964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:41.144964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:41.144964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.144964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:41.145321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:41.145636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:41.145972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:41.146458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:41.146458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:41.146458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:41.146458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:41.146458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:41.146458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.146458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:41.146458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:41.146458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.146458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:41.146458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.146458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:41.150355 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:41.150553 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:41.150634 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:41.150734 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:41.151041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.151265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.151616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:41.151616 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.151616 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:41.151616 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:41.151616 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.151616 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:41.151616 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.151616 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:41.152134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:41.152543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:41.152543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:41.152543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.152543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:41.152543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:41.152543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.152543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:41.152543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.152543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:41.152748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.152748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:41.152748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:41.152748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:41.152748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.152748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:41.152748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:41.152748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:41.152748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:41.152748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.152748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:41.153018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:41.153404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.153404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:41.153404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.153404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.153718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:41.155076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:41.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:41.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.155927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:41.166096 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:41.166568 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:41.166568 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:41.166699 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:41.166962 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.167237 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:41.167526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.168105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:41.168105 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.168105 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:41.168105 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:41.168105 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.168105 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:41.168105 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.168105 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:41.168985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.168985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:41.168985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:41.168985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:41.168985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.168985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:41.168985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:41.168985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.168985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:41.168985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:41.168985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:41.168985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.169329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:41.169640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.169640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:41.170169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:41.170838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:41.171483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:41.171483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:41.171483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:46.037674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:46.039055 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:46.039319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:46.040823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:46.105814 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.106140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.106879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:46.106879 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.106879 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:46.106879 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:46.106879 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.106879 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:46.106879 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.106879 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:46.107790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.067 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.107790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:46.107790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:46.107790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:46.107790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.107790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:46.107790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:46.107790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.107790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:46.107790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:46.107790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:46.107790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.108381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:46.109945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:46.110504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.111123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:46.111123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:46.111123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.111123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:46.111123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.111123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:46.133971 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:46.134370 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:46.134484 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:46.135828 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.136088 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.136279 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.136445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.137462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.137819 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.138062 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.138208 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.138313 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.138466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.139084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:46.139084 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.139084 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:46.139084 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:46.139084 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.139084 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:46.139084 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.139084 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:46.139903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.139903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:46.139903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:46.139903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:46.139903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.139903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:46.139903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:46.139903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.139903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:46.139903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:46.139903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:46.139903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.140374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:46.140899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:46.141335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.141815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:46.141815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:46.141815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.141815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:46.141815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.141815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:46.146468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:46.146692 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:46.146807 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:46.146898 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:05:46.148054 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.148322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.148956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:46.148956 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.148956 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:46.148956 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:46.148956 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.148956 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:46.148956 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.148956 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:46.150253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:46.150727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:46.150727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:46.150727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.150727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:46.150727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:46.150727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.150727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:46.150727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.150727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:46.152402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.152402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:46.152402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:46.152402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:46.152402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.152402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:46.152402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:46.152402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:05:46.152402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:46.152402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:46.152402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.152402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:46.152819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:46.153207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.153207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:46.153207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.153207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:46.153207 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:46.154775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:46.155281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:46.155770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.155770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:46.155770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:46.155770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.155770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:46.155770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.155770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:46.155770 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:46.155770 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:46.155770 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:46.156063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.156063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.157479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:46.157479 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.157479 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:46.157479 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:46.157479 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.157479 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:46.157479 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.157479 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:46.158130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:46.158581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:46.158581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:46.158581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.158581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:46.158581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:46.158581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.158581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:46.158581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.158581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:46.158824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.158824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:46.158824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:46.158824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:46.158824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.158824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:46.158824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:46.158824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:46.158824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:46.158824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.158824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:46.159159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:46.159531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.159531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:46.159531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.159531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.160830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:46.161500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:46.161988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.161988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:46.161988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.161988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:46.166546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:46.168307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:46.168307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:46.168307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:46.168307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.168307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:46.168538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.171464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:46.171464 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.171464 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:46.171464 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:46.171464 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.171464 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:46.171464 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.171464 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:46.172448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.172448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:46.172448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:46.172448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:46.172448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.172448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:46.172448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:46.172448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.172448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:46.172448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:46.172448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:46.172448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.172853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:46.173399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.173399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:46.175149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:46.175661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:46.176177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:46.176177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:46.176177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:51.039023 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:51.039599 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:51.039599 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:51.040108 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:51.069024 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.069269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:51.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:51.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:51.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:51.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.069749 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:51.070373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.070373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:51.070373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:51.070373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:51.070373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.070373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:51.070373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:51.070373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.070373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:51.070373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:51.070373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:51.070373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.070610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:51.070840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.070840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:51.071029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:51.071717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.072357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:51.072357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.072357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:51.134409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:51.134611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:51.134723 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:51.135633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.135787 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.135921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.136076 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.136198 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.136358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.136483 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.136617 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.136740 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.136871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.137405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:51.137405 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.137405 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:51.137405 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:51.137405 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.137405 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:51.137405 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.137405 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:51.137978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.137978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:51.137978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:51.137978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:51.137978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.137978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:51.137978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:51.137978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.137978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:51.137978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:51.137978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:51.137978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.138244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:51.138434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.138434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:51.138645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:51.138907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.139259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:51.139259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.139259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:51.146720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:51.146866 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:51.146959 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:51.147104 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:05:51.147388 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.147559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.147950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:51.147950 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.147950 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:51.147950 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:51.147950 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.147950 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:51.147950 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.147950 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:51.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:51.148753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:51.148753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:51.148753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.148753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:51.148753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:51.148753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.148753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:51.148753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.148753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:51.149049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.149049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:51.149049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:51.149049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:51.149049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.149049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:51.149049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:51.149049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:05:51.149049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:51.149049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:51.149049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.149049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:51.149454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:51.149971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:51.150305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:51.150623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:51.150623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:51.150623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:51.150623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:51.150623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:51.150623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.150623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:51.150623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:51.150623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.150623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:51.150623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.150623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:51.154989 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:51.155151 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:51.155268 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:51.155355 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:51.155605 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.155760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.156107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:51.156107 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.156107 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:51.156107 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:51.156107 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.156107 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:51.156107 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.156107 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:51.156639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:51.156912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:51.156912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:51.156912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.156912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:51.156912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:51.156912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.156912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:51.156912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.156912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:51.157323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.157323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:51.157323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:51.157323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:51.157323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.157323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:51.157323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:51.157323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:51.157323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:51.157323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.157323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:51.157629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:51.157947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:51.158253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:51.158536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:51.158536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:51.158536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:51.158536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:51.158536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.158536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:51.158536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:51.158536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.158536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:51.158536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.158536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:51.167052 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:51.167602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:51.167602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:51.167719 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:51.167960 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.168213 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:51.168485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:51.169993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:51.170488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.170712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:51.170941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:51.171294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:51.171294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:51.171294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:51.171294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:51.171294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:51.171294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:51.171294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:51.171294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:51.171294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:51.171294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:56.040132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:56.040554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:56.040584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:56.041206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:56.086452 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.086655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.087276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:56.087276 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.087276 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:56.087276 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:56.087276 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.087276 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:56.087276 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.087276 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:56.088252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.088252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:56.088252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:56.088252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:56.088252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.088252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:56.088252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:56.088252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.088252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:56.088252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:56.088252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:56.088252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.088528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:56.089107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.089107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:56.089409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:56.089693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.089994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:56.089994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.089994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:56.134818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:56.135132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:56.135328 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:56.136454 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.136627 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.136774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.136913 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.137071 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.137270 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.137368 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.137513 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.137649 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.137782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.138285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:56.138285 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.138285 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:56.138285 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:56.138285 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.138285 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:56.138285 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.138285 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:56.140015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.140015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:56.140015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:56.140015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:56.140015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.140015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:56.140015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:56.140015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.140015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:56.140015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:56.140015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:56.140015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.140408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:56.140727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:56.141175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.141713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:56.141713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:56.141713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.141713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:56.141713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.141713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:56.147112 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:56.147429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:56.147791 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:56.147791 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:05:56.148032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.148265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.150583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:56.151115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:56.151641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.151989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:56.152400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:56.152910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:56.153458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:56.153458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:56.153458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.153458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:56.153458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:56.153458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.153458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:56.153458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.153458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:56.155664 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:56.155664 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:56.155664 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:56.155664 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:56.155919 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.156154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.158330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:05:56.158675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:56.158954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.159236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:56.159447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:56.159719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:56.160158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:56.160158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.160158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:56.160158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:56.160158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.160158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:56.160158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.160158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:05:56.167601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:05:56.167838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:05:56.168169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:05:56.168169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:05:56.168446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.168596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:05:56.168859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.169299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:05:56.169299 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.169299 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:05:56.169299 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:05:56.169299 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.169299 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:05:56.169299 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.169299 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:05:56.175557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.175557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:05:56.175557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:56.175557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:05:56.175557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.175557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:56.175557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:56.175557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.175557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:56.175557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:56.175557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:05:56.175557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.175913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:05:56.176133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.176133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:05:56.176729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:05:56.177051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:05:56.177439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:05:56.177439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:05:56.177439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:01.041755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:01.041755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:01.041755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:01.043778 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:01.090655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.090815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.091260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:01.091260 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.091260 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:01.091260 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:01.091260 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.091260 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:01.091260 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.091260 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:01.091856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.091856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:01.091856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:01.091856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:01.091856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.091856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:01.091856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:01.091856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.091856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:01.091856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:01.091856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:01.091856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.092238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:01.092548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:01.092852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.093280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:01.093280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:01.093280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.093280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:01.093280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.093280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:01.135455 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:01.135658 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:01.135765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:01.136984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.137161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.137321 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.137457 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.137584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.137707 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.137834 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.137969 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.138114 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.138254 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.138384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.139164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:01.139164 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.139164 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:01.139164 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:01.139164 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.139164 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:01.139164 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.139164 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:01.140051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.140051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:01.140051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:01.140051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:01.140051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.140051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:01.140051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:01.140051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.140051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:01.140051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:01.140051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:01.140051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.140445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:01.140804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.140804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:01.141073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:01.141565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.142125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:01.142125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.142125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:01.147620 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:01.147890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:01.147991 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:01.148134 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:06:01.148488 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.148685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.149200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:01.149200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.149200 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:01.149200 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:01.149200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.149200 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:01.149200 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.149200 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:01.149914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:01.150477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:01.150477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:01.150477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.150477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:01.150477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:01.150477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.150477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:01.150477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.150477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:01.150870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.150870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:01.150870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:01.150870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:01.150870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.150870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:01.150870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:01.150870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:06:01.150870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:01.150870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:01.150870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.150870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:01.151339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:01.151704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.151704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:01.151704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.151704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:01.151898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:01.152431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:01.152980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.152980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:01.152980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:01.152980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.152980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:01.152980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.152980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:01.155613 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:01.155869 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:01.155970 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:01.156159 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:01.156488 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.156726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.157273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:01.157273 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.157273 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:01.157273 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:01.157273 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.157273 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:01.157273 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.157273 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:01.158001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:01.158468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:01.158468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:01.158468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.158468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:01.158468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:01.158468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.158468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:01.158468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.158468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:01.159129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.159129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:01.159129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:01.159129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:01.159129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.159129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:01.159129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:01.159129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:01.159129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:01.159129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.159129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:01.159535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:01.159945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.159945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:01.159945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.159945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.160223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:01.160725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:01.161311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.161311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:01.161311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.161311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:01.168060 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:01.168343 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:01.168572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:01.168753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:01.169102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.169361 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:01.169662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.170305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:01.170305 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.170305 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:01.170305 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:01.170305 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.170305 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:01.170305 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.170305 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:01.171530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.171530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:01.171530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:01.171530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:01.171530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.171530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:01.171530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:01.171530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.171530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:01.171530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:01.171530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:01.171530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.171945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:01.172308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.172308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:01.172757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:01.173131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:01.173471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:01.173471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:01.173471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:06.041788 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:06.042162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:06.042162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:06.042921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:06.085951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.085951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.086200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:06.086200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.086200 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:06.086200 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:06.086200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.086200 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:06.086200 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.086200 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:06.086951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.086951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:06.086951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:06.086951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:06.086951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.086951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:06.086951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:06.086951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.086951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:06.086951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:06.086951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:06.086951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.087217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:06.087453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.087453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:06.088012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:06.088353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.088674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:06.088674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.088674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:06.135811 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:06.136314 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:06.136314 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:06.140569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.142278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.142278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.142278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.142278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.142278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.142531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.142531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.142531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.142531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.142531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.142531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.142738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:06.142738 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.142738 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:06.142738 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:06.142738 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.142738 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:06.142738 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.142738 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:06.143670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.143670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:06.143670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:06.143670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:06.143670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.143670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:06.143670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:06.143670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.143670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:06.143670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:06.143670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:06.143670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.144090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:06.144495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.144495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:06.145058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:06.145407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.145845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:06.145845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.145845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:06.148016 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:06.148326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:06.148451 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:06.148574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:06:06.149558 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.149558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.149730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:06.149730 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.149730 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:06.149730 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:06.149730 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.149730 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:06.149730 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.149730 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:06.150381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:06.150655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:06.150655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:06.150655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.150655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:06.150655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:06.150655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.150655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:06.150655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.150655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:06.151119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.151119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:06.151119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:06.151119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:06.151119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.151119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:06.151119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:06.151119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:06:06.151119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:06.151119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:06.151119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.151119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:06.151386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:06.151591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.151591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:06.151591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.151591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:06.151882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:06.152155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:06.152437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.152437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:06.152437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:06.152437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.152437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:06.152437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.152437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:06.155940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:06.156206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:06.156309 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:06.156436 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:06.156804 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.157421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.157586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:06.157586 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.157586 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:06.157586 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:06.157586 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.157586 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:06.157586 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.157586 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:06.158299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:06.158577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:06.158577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:06.158577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.158577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:06.158577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:06.158577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.158577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:06.158577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.158577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:06.159048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.159048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:06.159048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:06.159048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:06.159048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.159048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:06.159048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:06.159048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:06.159048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:06.159048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.159048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:06.159282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:06.159515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.159515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:06.159515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.159515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.159976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:06.160363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:06.160706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.160706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:06.160706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.160706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:06.168459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:06.168712 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:06.168826 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:06.169008 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:06.170568 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.170568 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:06.170568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.170927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:06.170927 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.170927 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:06.170927 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:06.170927 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.170927 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:06.170927 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.170927 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:06.171921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.171921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:06.171921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:06.171921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:06.171921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.171921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:06.171921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:06.171921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.171921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:06.171921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:06.171921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:06.171921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.172348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:06.172574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.172574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:06.172882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:06.173336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:06.173869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:06.173869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:06.173869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:11.042170 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:11.042628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:11.042679 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:11.043328 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:11.302630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:11.314722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:11.315029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:11.315029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:11.356431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.356664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.358235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:11.358235 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.358235 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:11.358235 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:11.358235 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.358235 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:11.358235 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.358235 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:11.358597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.315 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.358597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:11.358597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:11.358597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:11.358597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.358597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:11.358597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:11.358597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.358597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:11.358597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:11.358597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:11.358597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.359049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:11.359440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.359440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:11.359700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:11.360167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.361372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:11.361372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.361372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:11.361372 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:11.361372 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:11.361564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.361807 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.361981 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.362630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.362630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.362630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.362630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.362870 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.362926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.363056 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.363235 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.363395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.364027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:11.364027 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.364027 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:11.364027 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:11.364027 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.364027 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:11.364027 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.364027 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:11.365141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.365141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:11.365141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:11.365141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:11.365141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.365141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:11.365141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:11.365141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.365141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:11.365141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:11.365141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:11.365141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.365509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:11.365742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.365742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:11.366807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:11.367501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.368294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:11.368294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.368294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:11.368294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:11.368294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:11.368294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:06:11.368683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.368903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.370460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:11.371937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:11.372491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.372869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:11.373550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:11.373980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:11.374687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:11.374687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:11.374687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.374687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:11.374687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:11.374687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.374687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:11.374687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.374687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:11.374687 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:11.374687 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:11.374687 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:11.375051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.375692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:11.379031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.379031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:11.379031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.379031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:11.379031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:11.379031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:11.379031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.379031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:11.379031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:11.379031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:11.379031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:11.379031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.379444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:11.379944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:11.380349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:11.381075 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:11.381462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:11.381462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.381462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:11.381462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.381462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:11.381462 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.381462 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:11.381462 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:11.381462 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.381462 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:11.381462 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.381462 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:11.381714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.381714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:11.381714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:11.381714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:11.381714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.381714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:11.381714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:11.381714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.381714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:11.381714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:11.381714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:11.381714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.381918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:11.382135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:11.382991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:11.382991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:11.382991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:11.382991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:11.382991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:11.382991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:16.042809 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:16.043135 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:16.043245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:16.043845 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:16.078414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.078641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.079289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:16.079289 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.079289 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:16.079289 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:16.079289 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.079289 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:16.079289 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.079289 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:16.080516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.080516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:16.080516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:16.080516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:16.080516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.080516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:16.080516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:16.080516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.080516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:16.080516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:16.080516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:16.080516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.080766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:16.081034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.081034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:16.081751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:16.082144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.082499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:16.082499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.082499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:16.361025 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:16.361025 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:16.361025 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:16.361697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.361876 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.362063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.362247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.362388 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.362535 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.362651 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.362801 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.362947 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.363127 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.363279 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.363437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.363960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:16.363960 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.363960 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:16.363960 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:16.363960 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.363960 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:16.363960 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.363960 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:16.364772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.364772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:16.364772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:16.364772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:16.364772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.364772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:16.364772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:16.364772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.364772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:16.364772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:16.364772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:16.364772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.365129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:16.365864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.365864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:16.365934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:16.366372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.366806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:16.366806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.366806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:16.366806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:16.366997 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:16.367114 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:16.367252 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:06:16.369942 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.370103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.370516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:16.370516 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.370516 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:16.370516 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:16.370516 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.370516 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:16.370516 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.370516 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:16.371212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:16.371613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:16.371613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:16.371613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.371613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:16.371613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:16.371613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.371613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:16.371613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.371613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:16.372411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.372411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:16.372411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:16.372411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:16.372411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.372411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:16.372411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:16.372411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:06:16.372411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:16.372411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:16.372411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.372411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:16.372737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:16.373059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.373059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:16.373059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.373059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:16.373156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:16.373525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:16.373928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.373928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:16.373928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:16.373928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.373928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:16.373928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.373928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:16.373928 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:16.374172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:16.374319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:16.374525 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:16.374815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.374988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.375333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:16.375333 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.375333 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:16.375333 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:16.375333 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.375333 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:16.375333 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.375333 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:16.376468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:16.376827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:16.376827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:16.376827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.376827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:16.376827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:16.376827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.376827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:16.376827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.376827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:16.377637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.377637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:16.377637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:16.377637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:16.377637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.377637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:16.377637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:16.377637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:16.377637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:16.377637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.377637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:16.377935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:16.378240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.378240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:16.378240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.378240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:16.378240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.378857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:16.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:16.379576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.379576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:16.379576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.379576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:16.380541 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:16.380630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:16.383235 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:16.383591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.383773 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:16.384014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.384531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:16.384531 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.384531 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:16.384531 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:16.384531 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.384531 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:16.384531 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.384531 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:16.385128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.385128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:16.385128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:16.385128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:16.385128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.385128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:16.385128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:16.385128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.385128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:16.385128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:16.385128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:16.385128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.385389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:16.385598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.385598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:16.385880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:16.386143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:16.386471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:16.386471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:16.386471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:21.043546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:21.044516 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:21.044608 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:21.045210 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:21.060984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=241 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:06:21.102656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.103300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.106730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:21.106730 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.106730 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:21.106730 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:21.106730 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.106730 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:21.106730 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.106730 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:21.108938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.062 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.108938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:21.108938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:21.108938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:21.108938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.108938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:21.108938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:21.108938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.108938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:21.108938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:21.108938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:21.108938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.109636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:21.109909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.109909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:21.110033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:21.110416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.110794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:21.110794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.110794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:21.360927 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:21.361281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:21.361281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:21.362671 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.362812 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.362991 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.363088 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.363227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.363344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.363421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.363560 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.363688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.363785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.363917 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.364021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.364677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:21.364677 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.364677 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:21.364677 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:21.364677 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.364677 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:21.364677 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.364677 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:21.365749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.365749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:21.365749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:21.365749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:21.365749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.365749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:21.365749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:21.365749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.365749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:21.365749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:21.365749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:21.365749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.366427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:21.366924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.366924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:21.367755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:21.368468 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.369628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:21.369628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.369628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:21.369628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:21.369628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:21.369628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:21.369628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:06:21.369628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.370053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.370331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:21.370331 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.370331 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:21.370331 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:21.370331 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.370331 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:21.370331 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.370331 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:21.371251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:21.374628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:21.375122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:21.375547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:21.375547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:21.375547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:21.375547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:21.375547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:21.375547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.375547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:21.375547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:21.375547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.375547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:21.375547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.375547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:21.375931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:21.376463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:21.377053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.377053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:21.377053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:21.377053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.377053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:21.377053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.377053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:21.377053 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:21.377053 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:21.377053 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:21.377429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:21.377429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.377429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.377631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:21.377631 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.377631 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:21.377631 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:21.377631 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.377631 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:21.377631 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.377631 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:21.378398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:21.378862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:21.378862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:21.378862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.378862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:21.378862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:21.378862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.378862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:21.378862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.378862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:21.379698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.379698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:21.379698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:21.379698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:21.379698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.379698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:21.379698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:21.379698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:21.379698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:21.379698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.379698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:21.380082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:21.380490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.380490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:21.380490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.380490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.380689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:21.381156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:21.381680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.381680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:21.381680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.381680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:21.381680 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:21.381680 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:21.381864 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:21.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:21.382241 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.382447 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:21.382714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.383106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:21.383106 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.383106 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:21.383106 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:21.383106 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.383106 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:21.383106 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.383106 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:21.384066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.384066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:21.384066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:21.384066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:21.384066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.384066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:21.384066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:21.384066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.384066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:21.384066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:21.384066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:21.384066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.384465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:21.384703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.384703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:21.385178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:21.385471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:21.385860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:21.385860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:21.385860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:24.158711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:24.159256 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:24.159256 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:24.173212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 09 01:06:24.173212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Number of inconsistencies found at create/update: networks=9, security_groups=2, routers=6, router_ports=6 {{(pid=89255) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 09 01:06:24.173212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=10 {{(pid=89255) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 09 01:06:24.173212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource b3e28628-ad68-4ca3-b181-b39c995a69e0 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.200528 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, resource={'id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'name': 'public', 'tenant_id': '5a148cfd27844ee6ad0d79cb077e4985', 'project_id': '5a148cfd27844ee6ad0d79cb077e4985', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['17135a44-456c-43be-8f7d-7714aa3731cf', 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-09T00:51:13Z', 'updated_at': '2026-06-09T00:51:20Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.200725 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.200961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-6a91ff68-89ba-411b-b54f-ff58910954a3, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.201216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.202364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.202364 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.202364 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.202364 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.202364 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.202364 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.202364 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.202364 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.202837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource b3e28628-ad68-4ca3-b181-b39c995a69e0 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.202837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.202837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.202837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.202837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.202837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.202837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:06:24.202837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.202837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.202837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.202837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.203133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.203405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.203405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.203405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.203405 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 05419c49-9db5-4d5d-8bbe-5a7016c12d7e (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.232275 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-05419c49-9db5-4d5d-8bbe-5a7016c12d7e, resource={'id': '05419c49-9db5-4d5d-8bbe-5a7016c12d7e', 'name': 'shared', 'tenant_id': '5a148cfd27844ee6ad0d79cb077e4985', 'project_id': '5a148cfd27844ee6ad0d79cb077e4985', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['612c5f15-747e-4ea8-a2d1-d0bbb8f1bc11'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:52:24Z', 'updated_at': '2026-06-09T00:52:25Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 44634}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.232462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-05419c49-9db5-4d5d-8bbe-5a7016c12d7e, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.232674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-1c288345-932b-445f-8f63-edb47405877c, external_ids_update=None, columns={'tag': 44634}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.232761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.233298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.233298 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.233298 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.233298 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.233298 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.233298 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.233298 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.233298 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.234397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 05419c49-9db5-4d5d-8bbe-5a7016c12d7e (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.234397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.234397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.234397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.234397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.234397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.234397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:06:24.234397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.234397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.234397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.234397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.234632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.234867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.234867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.234867 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.234867 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 444193f1-a412-4f84-ba64-43604e672d34 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.279161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-444193f1-a412-4f84-ba64-43604e672d34, resource={'id': '444193f1-a412-4f84-ba64-43604e672d34', 'name': 'tempest-TestVolumeBootPattern-967850104-network', 'tenant_id': '36dddccfbbfe427ebd19655d3040ba79', 'project_id': '36dddccfbbfe427ebd19655d3040ba79', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['af032e9a-f4ed-410c-aacd-5dbd586dd2b2'], 'standard_attr_id': 943, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:00:37Z', 'updated_at': '2026-06-09T01:00:38Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 6646}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.279545 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-444193f1-a412-4f84-ba64-43604e672d34, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPattern-967850104-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.279910 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-1390cd1f-7485-4133-8c6c-51a9fc2c35af, external_ids_update=None, columns={'tag': 6646}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.280857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.280857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.280857 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.280857 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.280857 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.280857 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.280857 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.280857 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.280857 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.282212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 444193f1-a412-4f84-ba64-43604e672d34 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.282212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.282212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.282212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.282212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.282212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.282212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:06:24.282212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.282212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.282212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.282212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.282572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.282897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.282897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.282897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.282897 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource dc15bda6-1b51-4a9e-af84-ac570d199c11 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.320183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-dc15bda6-1b51-4a9e-af84-ac570d199c11, resource={'id': 'dc15bda6-1b51-4a9e-af84-ac570d199c11', 'name': 'private', 'tenant_id': '5a148cfd27844ee6ad0d79cb077e4985', 'project_id': '5a148cfd27844ee6ad0d79cb077e4985', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['65e82083-c074-4bc8-b13e-2eeabecc299d', '6f142de2-8acf-46b2-af54-c34cf691eff5'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:51:08Z', 'updated_at': '2026-06-09T00:51:11Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 62028}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.320183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-dc15bda6-1b51-4a9e-af84-ac570d199c11, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.320418 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-77925902-9e1b-4f7c-868e-70a5d73ee785, external_ids_update=None, columns={'tag': 62028}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.320418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.320418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.320418 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.320418 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.320418 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.320418 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.320418 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.320418 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.320418 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.320616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource dc15bda6-1b51-4a9e-af84-ac570d199c11 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.320616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.320616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.320616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.320616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.320616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.320616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:06:24.320616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.320616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.320616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.320616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.320866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.321123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.321123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.321123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.321123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource f1f32932-fb0a-4be4-8616-2eabe6fcd858 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.358246 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f1f32932-fb0a-4be4-8616-2eabe6fcd858, resource={'id': 'f1f32932-fb0a-4be4-8616-2eabe6fcd858', 'name': 'tempest-TestVolumeBootPattern-1135803167-network', 'tenant_id': '34a3b55eac5948bca5d0dd729db545e0', 'project_id': '34a3b55eac5948bca5d0dd729db545e0', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['39bf8048-d927-46d7-ab5a-e5c0d784212a'], 'standard_attr_id': 955, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:00:45Z', 'updated_at': '2026-06-09T01:00:45Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 22541}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.358580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-f1f32932-fb0a-4be4-8616-2eabe6fcd858, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPattern-1135803167-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.358929 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-c0259406-a3e4-44cb-a638-ab378ee2d196, external_ids_update=None, columns={'tag': 22541}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.359077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.359788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.359788 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.359788 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.359788 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.359788 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.359788 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.359788 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.359788 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.360691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource f1f32932-fb0a-4be4-8616-2eabe6fcd858 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.360691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.360691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.360691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.360691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.360691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.360691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:06:24.360691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.360691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.360691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.360691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.360992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.361299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.361299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.361299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.361299 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 4c08284c-c041-4d26-999d-a1e7eae23340 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.394280 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-4c08284c-c041-4d26-999d-a1e7eae23340, resource={'id': '4c08284c-c041-4d26-999d-a1e7eae23340', 'name': 'tempest-VolumesBackupsTest-1891334833-network', 'tenant_id': '80666ebd43dc4980975b7dedfe19fe2f', 'project_id': '80666ebd43dc4980975b7dedfe19fe2f', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['2e1de07b-2fea-42c1-8056-f6464a1fc8aa'], 'standard_attr_id': 1291, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:04:57Z', 'updated_at': '2026-06-09T01:04:58Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 28016}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.394543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-4c08284c-c041-4d26-999d-a1e7eae23340, col_values=(('external_ids', {'neutron:network_name': 'tempest-VolumesBackupsTest-1891334833-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.394717 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-0317dbec-d217-4597-8cc7-5040ef916ec5, external_ids_update=None, columns={'tag': 28016}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.394825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.396264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.396264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.396264 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.396264 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.396264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.396264 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.396264 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.396264 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.397527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 4c08284c-c041-4d26-999d-a1e7eae23340 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.397527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.397527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.397527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.397527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.397527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.397527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:06:24.397527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.397527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.397527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.397527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.397937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.398306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.398306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.398306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.398306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 9cd7a1a5-7202-4795-9d4c-244d29bdd4b6 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.426523 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-9cd7a1a5-7202-4795-9d4c-244d29bdd4b6, resource={'id': '9cd7a1a5-7202-4795-9d4c-244d29bdd4b6', 'name': 'tempest-TestMinimumBasicScenario-2037712642-network', 'tenant_id': 'ceac60e2e7544e9da42f88a6a2301c2e', 'project_id': 'ceac60e2e7544e9da42f88a6a2301c2e', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['f4c1d176-f5dc-4a84-90ca-444a036e30a6'], 'standard_attr_id': 1216, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:04:24Z', 'updated_at': '2026-06-09T01:04:24Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 1146}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.426523 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-9cd7a1a5-7202-4795-9d4c-244d29bdd4b6, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestMinimumBasicScenario-2037712642-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.426779 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-557ac3b8-6878-4451-bd0b-fb221035efea, external_ids_update=None, columns={'tag': 1146}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.426779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.426779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.426779 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.426779 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.426779 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.426779 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.426779 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.426779 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.426779 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.427655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 9cd7a1a5-7202-4795-9d4c-244d29bdd4b6 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.427655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.427655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.427655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.427655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.427655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.427655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:06:24.427655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.427655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.427655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.427655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.427911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.428140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.428140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.428140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.428140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource fa41affb-e292-4ef8-ba22-87027459ecbe (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.452530 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-fa41affb-e292-4ef8-ba22-87027459ecbe, resource={'id': 'fa41affb-e292-4ef8-ba22-87027459ecbe', 'name': 'tempest-VolumesSnapshotTestJSON-1317468788-network', 'tenant_id': '962f01b983f74a5a9fd7a8daeb006cfd', 'project_id': '962f01b983f74a5a9fd7a8daeb006cfd', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['1d46220e-6a25-4b9a-8f4a-293e872a7c20'], 'standard_attr_id': 1146, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:03:48Z', 'updated_at': '2026-06-09T01:03:49Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 59324}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.452805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-fa41affb-e292-4ef8-ba22-87027459ecbe, col_values=(('external_ids', {'neutron:network_name': 'tempest-VolumesSnapshotTestJSON-1317468788-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.453105 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-0826806f-c424-43d0-a41a-ed5d16c9851a, external_ids_update=None, columns={'tag': 59324}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.453247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.453891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.453891 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.453891 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.453891 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.453891 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.453891 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.453891 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.453891 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.454761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource fa41affb-e292-4ef8-ba22-87027459ecbe (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.454761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.454761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.454761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.454761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.454761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.454761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:06:24.454761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.454761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.454761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.454761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.455111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.455484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.455484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.455484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.455484 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 6b4f028f-e8d4-4eb7-b5d8-0e576bc9c0ba (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.479211 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-6b4f028f-e8d4-4eb7-b5d8-0e576bc9c0ba, resource={'id': '6b4f028f-e8d4-4eb7-b5d8-0e576bc9c0ba', 'name': 'tempest-TestMinimumBasicScenario-1388690340-network', 'tenant_id': '6e4618f194fe4db790ed4295d2259579', 'project_id': '6e4618f194fe4db790ed4295d2259579', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['7cd865b1-b88b-4132-8e38-69bebc025cc2'], 'standard_attr_id': 1203, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:04:18Z', 'updated_at': '2026-06-09T01:04:18Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 6584}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.479519 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-6b4f028f-e8d4-4eb7-b5d8-0e576bc9c0ba, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestMinimumBasicScenario-1388690340-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.479815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-16d01225-fec1-46ca-9479-b8037bf98ec0, external_ids_update=None, columns={'tag': 6584}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.479941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.480683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.480683 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.480683 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.480683 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.480683 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.480683 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.480683 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.480683 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.484247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 6b4f028f-e8d4-4eb7-b5d8-0e576bc9c0ba (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.484247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.484247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.484247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.484247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.484247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.484247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:06:24.484247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.484247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.484247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.484247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.484524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.484760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.484760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.484760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.484760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 253e512d-553a-40f3-bfd8-2b5f4dfdd9bd (type: security_groups) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.506170 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Successfully bumped revision number for resource 253e512d-553a-40f3-bfd8-2b5f4dfdd9bd (type: security_groups) to 3 Jun 09 01:06:24.506354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource d45a9922-e36f-4bc2-9732-960fe3c18539 (type: security_groups) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.529338 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Successfully bumped revision number for resource d45a9922-e36f-4bc2-9732-960fe3c18539 (type: security_groups) to 3 Jun 09 01:06:24.529477 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.597868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, resource={'id': 'f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2', 'name': 'tempest-VolumesSnapshotTestJSON-1317468788-router', 'project_id': '962f01b983f74a5a9fd7a8daeb006cfd', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.43'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::175'}], 'enable_snat': True}, 'gw_port_id': '918511a3-ffa0-4e0c-ab1b-367d461fbb3d', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.43', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::175', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:03:49Z', 'updated_at': '2026-06-09T01:03:51Z', 'revision_number': 3, 'tenant_id': '962f01b983f74a5a9fd7a8daeb006cfd'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.598129 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, columns={'external_ids': {'neutron:router_name': 'tempest-VolumesSnapshotTestJSON-1317468788-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.598382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "918511a3-ffa0-4e0c-ab1b-367d461fbb3d"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.598546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "918511a3-ffa0-4e0c-ab1b-367d461fbb3d"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.598683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.599484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.599484 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.599484 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.599484 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.599484 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.599484 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.599484 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.599484 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.599779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.600779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.600779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.600779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.600779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.600779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.600779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.600779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:06:24.600779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:06:24.600779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:06:24.600779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:06:24.600779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.601141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.601511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.601511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.601511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.601511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.601511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.601511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.601511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.601511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.601511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.601511 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 6de85ce9-38ca-4cad-b472-ee3299d66135 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.662171 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, resource={'id': '6de85ce9-38ca-4cad-b472-ee3299d66135', 'name': 'tempest-TestMinimumBasicScenario-1388690340-router', 'project_id': '6e4618f194fe4db790ed4295d2259579', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.135'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::f9'}], 'enable_snat': True}, 'gw_port_id': '88aa1dbf-106a-445b-8e36-c1bd3858be24', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.135', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::f9', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:04:19Z', 'updated_at': '2026-06-09T01:04:22Z', 'revision_number': 3, 'tenant_id': '6e4618f194fe4db790ed4295d2259579'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.662429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, columns={'external_ids': {'neutron:router_name': 'tempest-TestMinimumBasicScenario-1388690340-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.662595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "88aa1dbf-106a-445b-8e36-c1bd3858be24"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.662719 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "88aa1dbf-106a-445b-8e36-c1bd3858be24"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.662807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.663368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.663368 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.663368 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.663368 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.663368 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.663368 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.663368 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.663368 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.663558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 6de85ce9-38ca-4cad-b472-ee3299d66135. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.664544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 6de85ce9-38ca-4cad-b472-ee3299d66135 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.664544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.664544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.664544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.664544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.664544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.664544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:06:24.664544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:06:24.664544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:06:24.664544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:06:24.664544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.664737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.664967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.664967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.664967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.664967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.664967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.664967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.664967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.664967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.664967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.664967 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 1fb18b94-7a38-4aab-9569-672e24d0c07f (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.730801 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, resource={'id': '1fb18b94-7a38-4aab-9569-672e24d0c07f', 'name': 'tempest-TestMinimumBasicScenario-2037712642-router', 'project_id': 'ceac60e2e7544e9da42f88a6a2301c2e', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.220'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::147'}], 'enable_snat': True}, 'gw_port_id': '23f291dc-efcc-4517-a355-ee38f017fdde', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.220', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::147', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:04:25Z', 'updated_at': '2026-06-09T01:04:27Z', 'revision_number': 3, 'tenant_id': 'ceac60e2e7544e9da42f88a6a2301c2e'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.731079 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, columns={'external_ids': {'neutron:router_name': 'tempest-TestMinimumBasicScenario-2037712642-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.731296 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "23f291dc-efcc-4517-a355-ee38f017fdde"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.731426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "23f291dc-efcc-4517-a355-ee38f017fdde"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.731520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.732102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.732102 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.732102 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.732102 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.732102 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.732102 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.732102 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.732102 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.732294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 1fb18b94-7a38-4aab-9569-672e24d0c07f. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.733311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 1fb18b94-7a38-4aab-9569-672e24d0c07f (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.733311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.733311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.733311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.733311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.733311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.733311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:06:24.733311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:06:24.733311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:06:24.733311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:06:24.733311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.733524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.733753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.733753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.733753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.733753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.733753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.733753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.733753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.733753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.733753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.733753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 5feabefa-bba2-4b41-a481-a9141d275106 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.802338 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5feabefa-bba2-4b41-a481-a9141d275106, resource={'id': '5feabefa-bba2-4b41-a481-a9141d275106', 'name': 'tempest-TestVolumeBootPattern-1135803167-router', 'project_id': '34a3b55eac5948bca5d0dd729db545e0', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.78'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::20c'}], 'enable_snat': True}, 'gw_port_id': '4abb940c-0e56-4258-9c51-4bebfb8b2751', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.78', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::20c', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:00:46Z', 'updated_at': '2026-06-09T01:00:48Z', 'revision_number': 3, 'tenant_id': '34a3b55eac5948bca5d0dd729db545e0'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.802568 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-5feabefa-bba2-4b41-a481-a9141d275106, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPattern-1135803167-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.802751 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "4abb940c-0e56-4258-9c51-4bebfb8b2751"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.802873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "4abb940c-0e56-4258-9c51-4bebfb8b2751"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.802981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.803546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.803546 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.803546 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.803546 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.803546 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.803546 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.803546 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.803546 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.803719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 5feabefa-bba2-4b41-a481-a9141d275106. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.804769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 5feabefa-bba2-4b41-a481-a9141d275106 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.804769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.804769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.804769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.804769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.804769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.804769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:06:24.804769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:06:24.804769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:06:24.804769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:06:24.804769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.804970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.805257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.805257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.805257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.805257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.805257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.805257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.805257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.805257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.805257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.805257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 8e56abc3-0f43-47f4-9602-66084910993c (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.878374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-8e56abc3-0f43-47f4-9602-66084910993c, resource={'id': '8e56abc3-0f43-47f4-9602-66084910993c', 'name': 'tempest-VolumesBackupsTest-1891334833-router', 'project_id': '80666ebd43dc4980975b7dedfe19fe2f', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.196'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::29a'}], 'enable_snat': True}, 'gw_port_id': '4cef3b97-e0f2-4bb0-ad37-19f677f1f6ef', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.196', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::29a', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:04:59Z', 'updated_at': '2026-06-09T01:05:01Z', 'revision_number': 3, 'tenant_id': '80666ebd43dc4980975b7dedfe19fe2f'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.878642 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-8e56abc3-0f43-47f4-9602-66084910993c, columns={'external_ids': {'neutron:router_name': 'tempest-VolumesBackupsTest-1891334833-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.878861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "4cef3b97-e0f2-4bb0-ad37-19f677f1f6ef"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.879028 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "4cef3b97-e0f2-4bb0-ad37-19f677f1f6ef"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.879162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.879871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.879871 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.879871 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.879871 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.879871 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.879871 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.879871 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.879871 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.880034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 8e56abc3-0f43-47f4-9602-66084910993c. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.881325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 8e56abc3-0f43-47f4-9602-66084910993c (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.881325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.881325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.881325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.881325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.881325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.881325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:06:24.881325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:06:24.881325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:06:24.881325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:06:24.881325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.881547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.881831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.881831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.881831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.881831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.881831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.881831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.881831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.881831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.881831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.881831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 08c16dd9-f491-44d4-963d-2a87525d5e42 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:24.957353 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, resource={'id': '08c16dd9-f491-44d4-963d-2a87525d5e42', 'name': 'tempest-TestVolumeBootPattern-967850104-router', 'project_id': '36dddccfbbfe427ebd19655d3040ba79', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.176'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::190'}], 'enable_snat': True}, 'gw_port_id': 'bb3d4216-6a56-4092-9ac7-8a89e97dc07f', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.176', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::190', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:00:38Z', 'updated_at': '2026-06-09T01:00:42Z', 'revision_number': 3, 'tenant_id': '36dddccfbbfe427ebd19655d3040ba79'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.957729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPattern-967850104-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.957926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "bb3d4216-6a56-4092-9ac7-8a89e97dc07f"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.958071 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "bb3d4216-6a56-4092-9ac7-8a89e97dc07f"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:24.958171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.958975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:24.958975 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.958975 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:24.958975 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:24.958975 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.958975 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:24.958975 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.958975 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:24.959220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 08c16dd9-f491-44d4-963d-2a87525d5e42. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.960142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 08c16dd9-f491-44d4-963d-2a87525d5e42 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.960142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:24.960142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:24.960142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:24.960142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:24.960142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:24.960142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:06:24.960142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:06:24.960142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:06:24.960142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:06:24.960142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:24.960486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:24.960736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:24.960736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:24.960736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:24.960736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:24.960736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:24.960736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:24.960736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:24.960736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:24.960736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:24.960736 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 918511a3-ffa0-4e0c-ab1b-367d461fbb3d (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:25.042554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-918511a3-ffa0-4e0c-ab1b-367d461fbb3d, resource={'id': '918511a3-ffa0-4e0c-ab1b-367d461fbb3d', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:97:35:9a', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1151, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.43'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::175'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:03:49Z', 'updated_at': '2026-06-09T01:03:51Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.042932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-918511a3-ffa0-4e0c-ab1b-367d461fbb3d, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.43/24', '2001:db8::175/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.043297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=918511a3-ffa0-4e0c-ab1b-367d461fbb3d, lrouter_port=lrp-918511a3-ffa0-4e0c-ab1b-367d461fbb3d, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.043535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.044184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:25.044184 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:25.044184 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:25.044184 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:25.044184 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:25.044184 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:25.044184 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.044184 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:25.045115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 918511a3-ffa0-4e0c-ab1b-367d461fbb3d (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.045115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:25.045115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:25.045115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:25.045115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:25.045115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:25.045115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:06:25.045115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:25.045115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:25.045115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:25.045115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:25.045551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:25.045950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:25.045950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.045950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:25.045950 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 88aa1dbf-106a-445b-8e36-c1bd3858be24 (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:25.126608 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-88aa1dbf-106a-445b-8e36-c1bd3858be24, resource={'id': '88aa1dbf-106a-445b-8e36-c1bd3858be24', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:20:7d:07', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '6de85ce9-38ca-4cad-b472-ee3299d66135', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1208, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.135'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::f9'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:04:19Z', 'updated_at': '2026-06-09T01:04:22Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.126818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-88aa1dbf-106a-445b-8e36-c1bd3858be24, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-6de85ce9-38ca-4cad-b472-ee3299d66135'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::f9/64', '172.24.5.135/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.127043 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=88aa1dbf-106a-445b-8e36-c1bd3858be24, lrouter_port=lrp-88aa1dbf-106a-445b-8e36-c1bd3858be24, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.127231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.127660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:25.127660 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:25.127660 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:25.127660 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:25.127660 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:25.127660 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:25.127660 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.127660 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:25.128490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 88aa1dbf-106a-445b-8e36-c1bd3858be24 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.128490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:25.128490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:25.128490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:25.128490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:25.128490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:25.128490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:06:25.128490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:25.128490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:25.128490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:25.128490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:25.128842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:25.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:25.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:25.129272 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 4abb940c-0e56-4258-9c51-4bebfb8b2751 (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:25.208465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-4abb940c-0e56-4258-9c51-4bebfb8b2751, resource={'id': '4abb940c-0e56-4258-9c51-4bebfb8b2751', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:60:b5:74', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '5feabefa-bba2-4b41-a481-a9141d275106', 'device_owner': 'network:router_gateway', 'standard_attr_id': 960, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.78'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::20c'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:00:46Z', 'updated_at': '2026-06-09T01:00:49Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.209034 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-4abb940c-0e56-4258-9c51-4bebfb8b2751, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-5feabefa-bba2-4b41-a481-a9141d275106'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::20c/64', '172.24.5.78/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.209137 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=4abb940c-0e56-4258-9c51-4bebfb8b2751, lrouter_port=lrp-4abb940c-0e56-4258-9c51-4bebfb8b2751, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.209354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.209966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:25.209966 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:25.209966 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:25.209966 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:25.209966 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:25.209966 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:25.209966 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.209966 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:25.211151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 4abb940c-0e56-4258-9c51-4bebfb8b2751 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.211151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:25.211151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:25.211151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:25.211151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:25.211151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:25.211151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:06:25.211151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:25.211151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:25.211151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:25.211151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:25.211521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:25.211926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:25.211926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.211926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:25.211926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 23f291dc-efcc-4517-a355-ee38f017fdde (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:25.309508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-23f291dc-efcc-4517-a355-ee38f017fdde, resource={'id': '23f291dc-efcc-4517-a355-ee38f017fdde', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:ac:f1:da', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '1fb18b94-7a38-4aab-9569-672e24d0c07f', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1221, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.220'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::147'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:04:25Z', 'updated_at': '2026-06-09T01:04:30Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.309754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-23f291dc-efcc-4517-a355-ee38f017fdde, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.220/24', '2001:db8::147/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.309999 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=23f291dc-efcc-4517-a355-ee38f017fdde, lrouter_port=lrp-23f291dc-efcc-4517-a355-ee38f017fdde, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.310209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.310791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:25.310791 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:25.310791 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:25.310791 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:25.310791 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:25.310791 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:25.310791 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.310791 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:25.311715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 23f291dc-efcc-4517-a355-ee38f017fdde (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.311715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:25.311715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:25.311715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:25.311715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:25.311715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:25.311715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:06:25.311715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:25.311715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:25.311715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:25.311715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:25.312007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:25.312291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:25.312291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.312291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:25.312291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 4cef3b97-e0f2-4bb0-ad37-19f677f1f6ef (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:25.408078 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-4cef3b97-e0f2-4bb0-ad37-19f677f1f6ef, resource={'id': '4cef3b97-e0f2-4bb0-ad37-19f677f1f6ef', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:60:04:16', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '8e56abc3-0f43-47f4-9602-66084910993c', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1301, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.196'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::29a'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:04:59Z', 'updated_at': '2026-06-09T01:05:02Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.408469 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-4cef3b97-e0f2-4bb0-ad37-19f677f1f6ef, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-8e56abc3-0f43-47f4-9602-66084910993c'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.196/24', '2001:db8::29a/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.409808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=4cef3b97-e0f2-4bb0-ad37-19f677f1f6ef, lrouter_port=lrp-4cef3b97-e0f2-4bb0-ad37-19f677f1f6ef, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.413145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.413433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:25.413433 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:25.413433 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:25.413433 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:25.413433 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:25.413433 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:25.413433 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.413433 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:25.414556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 4cef3b97-e0f2-4bb0-ad37-19f677f1f6ef (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.414556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:25.414556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:25.414556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:25.414556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:25.414556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:25.414556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:06:25.414556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:25.414556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:25.414556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:25.414556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:25.414785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:25.414992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:25.414992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.414992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:25.414992 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource bb3d4216-6a56-4092-9ac7-8a89e97dc07f (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:06:25.524549 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-bb3d4216-6a56-4092-9ac7-8a89e97dc07f, resource={'id': 'bb3d4216-6a56-4092-9ac7-8a89e97dc07f', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:e1:da:ce', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '08c16dd9-f491-44d4-963d-2a87525d5e42', 'device_owner': 'network:router_gateway', 'standard_attr_id': 948, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.176'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::190'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:00:38Z', 'updated_at': '2026-06-09T01:00:41Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.524699 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-bb3d4216-6a56-4092-9ac7-8a89e97dc07f, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-08c16dd9-f491-44d4-963d-2a87525d5e42'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::190/64', '172.24.5.176/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.524699 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=bb3d4216-6a56-4092-9ac7-8a89e97dc07f, lrouter_port=lrp-bb3d4216-6a56-4092-9ac7-8a89e97dc07f, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:25.524699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.524829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:25.524829 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:25.524829 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:25.524829 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:25.524829 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:25.524829 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:25.524829 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.524829 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:25.525647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource bb3d4216-6a56-4092-9ac7-8a89e97dc07f (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.525647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:25.525647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:06:25.525647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:06:25.525647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:06:25.525647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:06:25.525647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:06:25.525647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:25.525647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:25.525647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:25.525647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:25.525947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:25.526350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:25.526350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:25.526350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:25.526350 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 175878c3-ae5a-4210-859f-45696c6b4999 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:06:25.526350 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(175878c3-ae5a-4210-859f-45696c6b4999) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:06:25.530951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 1b6c49ea-8289-449e-9c47-39f104796dd1 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:06:25.531551 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(1b6c49ea-8289-449e-9c47-39f104796dd1) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:06:25.536452 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 4a8b370e-f68f-43ed-939a-0f29e8a361f2 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:06:25.536746 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(4a8b370e-f68f-43ed-939a-0f29e8a361f2) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:06:25.541872 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 95bf70ca-58e3-4fae-a612-fe561367e0cc (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:06:25.542030 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(95bf70ca-58e3-4fae-a612-fe561367e0cc) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:06:25.545736 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource dad22407-98da-463e-b319-a4d46802b839 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:06:25.545948 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(dad22407-98da-463e-b319-a4d46802b839) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:06:25.549872 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource e76aa56f-1a2b-45a0-99fc-d9972b2bc0b1 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:06:25.550069 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(e76aa56f-1a2b-45a0-99fc-d9972b2bc0b1) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:06:25.554552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource ec1323ed-b9b5-4b00-a74e-fe7ee50211ed (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:06:25.554735 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(ec1323ed-b9b5-4b00-a74e-fe7ee50211ed) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:06:25.558399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource f1af8c78-c218-4a6e-adbf-080a8cb5c241 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:06:25.558523 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(f1af8c78-c218-4a6e-adbf-080a8cb5c241) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:06:25.562376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource f612d464-2a07-4d0d-8d1b-99a5d66c807b (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:06:25.562572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(f612d464-2a07-4d0d-8d1b-99a5d66c807b) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:06:25.568870 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource f6dc70fb-7c27-48ec-8719-bffba1fc90fe (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:06:25.569154 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(f6dc70fb-7c27-48ec-8719-bffba1fc90fe) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:06:25.574586 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_for_inconsistencies finished in 1.415 seconds Jun 09 01:06:26.045455 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:26.045793 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:26.045922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:26.046569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:26.081437 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.081437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.081677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:26.081677 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.081677 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:26.081677 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:26.081677 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.081677 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:26.081677 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.081677 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:26.082327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.082327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:26.082327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:26.082327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:26.082327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.082327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:26.082327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:26.082327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.082327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:26.082327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:26.082327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:26.082327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.083106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:26.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:26.084674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.085772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:26.085772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:26.085772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.085772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:26.085772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.085772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:26.361283 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:26.361692 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:26.361692 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:26.364806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.364806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.364806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.364806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.364806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.365019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.365019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.365019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.365019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.365019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.365147 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.365147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.365147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:26.365147 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.365147 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:26.365147 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:26.365147 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.365147 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:26.365147 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.365147 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:26.365715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.365715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:26.365715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:26.365715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:26.365715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.365715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:26.365715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:26.365715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.365715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:26.365715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:26.365715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:26.365715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.365984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:26.366212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.366212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:26.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:26.366734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.367012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:26.367012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.367012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:26.368545 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:26.368807 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:26.368888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:26.368983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:06:26.369808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.369808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.369938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:26.369938 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.369938 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:26.369938 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:26.369938 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.369938 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:26.369938 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.369938 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:26.370539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:26.371024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:26.371024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:26.371024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.371024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:26.371024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:26.371024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.371024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:26.371024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.371024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:26.371353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.371353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:26.371353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:26.371353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:26.371353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.371353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:26.371353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:26.371353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:06:26.371353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:26.371353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:26.371353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.371353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:26.371736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:26.372256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:26.372699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:26.373434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:26.373434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:26.373434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:26.373434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:26.373434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:26.373434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.373434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:26.373434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:26.373434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.373434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:26.373434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.373434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:26.374501 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:26.374758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:26.374870 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:26.374997 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:26.375820 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.375820 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.375956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:26.375956 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.375956 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:26.375956 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:26.375956 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.375956 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:26.375956 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.375956 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:26.376626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:26.377113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:26.377113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:26.377113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.377113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:26.377113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:26.377113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.377113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:26.377113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.377113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:26.377416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.377416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:26.377416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:26.377416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:26.377416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.377416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:26.377416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:26.377416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:26.377416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:26.377416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.377416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:26.377890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:26.378376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:26.378858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:26.379317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:26.379317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:26.379317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:26.379317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:26.379317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.379317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:26.379317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:26.379317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.379317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:26.379317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.379317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:26.381309 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:26.381643 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:26.381748 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:26.381883 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:26.382348 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.382647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:26.382985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.383521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:26.383521 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.383521 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:26.383521 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:26.383521 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.383521 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:26.383521 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.383521 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:26.384357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.384357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:26.384357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:26.384357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:26.384357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.384357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:26.384357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:26.384357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.384357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:26.384357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:26.384357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:26.384357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.384789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:26.385235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:26.385486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:26.385780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:26.385780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:26.385780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:26.385780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:26.385780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:26.385780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:31.046559 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:31.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:31.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:31.048561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:31.088073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.088205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:31.090236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:31.090607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.090905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:31.091657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:31.092233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:31.092233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:31.092233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.092233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:31.092233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:31.092233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.092233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:31.092233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.092233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:31.361747 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:31.362935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:31.362935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:31.363286 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.363465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.363619 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.363773 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.364030 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.364180 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.364343 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.364481 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.364625 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.364763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.364903 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.365158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:31.367238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:31.367636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.367978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:31.368303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:31.368663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:31.368663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:31.368663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:31.368663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.368663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:31.368663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:31.368663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.368663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:31.368663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.368663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:31.368944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:31.369250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:31.369421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:31.369590 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:06:31.370171 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.370363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.372853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:31.373316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:31.373746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.374001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:31.374358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:31.374684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:31.375126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:31.375126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:31.375126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.375126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:31.375126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:31.375126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.375126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:31.375126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.375126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:31.375126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:31.375510 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:31.375694 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:31.375971 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:31.376502 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.376708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.379179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:31.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:31.379934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.380174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:31.380497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:31.380821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:31.381272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:31.381272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.381272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:31.381272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:31.381272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.381272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:31.381272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.381272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:31.381596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:31.381793 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:31.381954 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:31.382168 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:31.382562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.382734 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:31.382978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:31.384650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:31.384941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.385236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:31.385489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:31.385802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:31.385802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:31.385802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:31.385802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:31.385802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:31.385802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:31.385802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:31.385802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:31.385802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:31.385802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:36.047278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:36.047674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:36.047674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:36.049145 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:36.085335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.085515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.086264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:36.086264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.086264 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:36.086264 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:36.086264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.086264 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:36.086264 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.086264 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:36.086974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.086974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:36.086974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:36.086974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:36.086974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.086974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:36.086974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:36.086974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.086974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:36.086974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:36.086974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:36.086974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.087286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:36.087528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.087528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:36.088049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:36.088376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.088738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:36.088738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.088738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:36.362274 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:36.364615 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:36.364615 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:36.365576 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.365728 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.365857 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.365956 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.366063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.366183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.366330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.366420 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.366514 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.366606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.366693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.366787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.367201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:36.367201 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.367201 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:36.367201 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:36.367201 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.367201 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:36.367201 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.367201 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:36.367852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.367852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:36.367852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:36.367852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:36.367852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.367852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:36.367852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:36.367852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.367852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:36.367852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:36.367852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:36.367852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.368098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:36.368369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.368369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:36.368643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:36.368922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.369313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:36.369313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.369313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:36.369313 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:36.369522 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:36.369522 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:36.369635 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:06:36.369823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.369998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.370339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:36.370339 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.370339 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:36.370339 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:36.370339 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.370339 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:36.370339 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.370339 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:36.370861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:36.371161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:36.371161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:36.371161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.371161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:36.371161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:36.371161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.371161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:36.371161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.371161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:36.371425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.371425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:36.371425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:36.371425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:36.371425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.371425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:36.371425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:36.371425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:06:36.371425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:36.371425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:36.371425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.371425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:36.371721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:36.371946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.371946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:36.371946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.371946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:36.372163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:36.372477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:36.372802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.372802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:36.372802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:36.372802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.372802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:36.372802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.372802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:36.375564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:36.375726 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:36.375835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:36.375970 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:36.376516 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.376689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.377152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:36.377152 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.377152 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:36.377152 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:36.377152 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.377152 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:36.377152 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.377152 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:36.377918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:36.378392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:36.378392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:36.378392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.378392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:36.378392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:36.378392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.378392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:36.378392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.378392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:36.378869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.378869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:36.378869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:36.378869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:36.378869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.378869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:36.378869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:36.378869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:36.378869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:36.378869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.378869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:36.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:36.379613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.379613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:36.379613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.379613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.380014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:36.380540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:36.381061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.381061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:36.381061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.381061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:36.382021 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:36.382338 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:36.382446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:36.382667 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:36.383010 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.383490 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:36.383897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.384487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:36.384487 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.384487 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:36.384487 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:36.384487 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.384487 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:36.384487 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.384487 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:36.385589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.385589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:36.385589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:36.385589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:36.385589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.385589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:36.385589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:36.385589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.385589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:36.385589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:36.385589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:36.385589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.385946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:36.386340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.386340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:36.386793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:36.387250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:36.387723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:36.387723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:36.387723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:41.048329 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:41.048774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:41.048774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:41.049297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:41.096452 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.096660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:41.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:41.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:41.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:41.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.097225 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:41.097894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.097894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:41.097894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:41.097894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:41.097894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.097894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:41.097894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:41.097894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.097894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:41.097894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:41.097894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:41.097894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.098204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:41.098563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.098563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:41.098946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:41.099338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.099743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:41.099743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.099743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:41.365429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:41.365701 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:41.365815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:41.368367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.368367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.368367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.368367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.368367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.368554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.368554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.368554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.368554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.368554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.368695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.368695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.368695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:41.368695 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.368695 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:41.368695 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:41.368695 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.368695 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:41.368695 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.368695 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:41.369343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.369343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:41.369343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:41.369343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:41.369343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.369343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:41.369343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:41.369343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.369343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:41.369343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:41.369343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:41.369343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.369586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:41.369790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.369790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:41.370474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:41.370751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.371073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:41.371073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.371073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:41.371073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:41.371073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:41.371073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:41.371073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:06:41.371750 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.371750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.371866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:41.371866 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.371866 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:41.371866 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:41.371866 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.371866 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:41.371866 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.371866 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:41.372587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:41.372875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:41.372875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:41.372875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.372875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:41.372875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:41.372875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.372875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:41.372875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.372875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:41.373383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.373383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:41.373383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:41.373383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:41.373383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.373383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:41.373383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:41.373383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:06:41.373383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:41.373383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:41.373383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.373383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:41.373611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:41.373837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.373837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:41.373837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.373837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:41.374354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:41.374629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:41.374914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.374914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:41.374914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:41.374914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.374914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:41.374914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.374914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:41.375772 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:41.375942 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:41.376060 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:41.376171 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:41.376750 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.376750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.376818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:41.376818 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.376818 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:41.376818 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:41.376818 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.376818 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:41.376818 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.376818 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:41.377532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:41.377873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:41.377873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:41.377873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.377873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:41.377873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:41.377873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.377873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:41.377873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.377873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:41.378408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.378408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:41.378408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:41.378408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:41.378408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.378408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:41.378408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:41.378408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:41.378408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:41.378408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.378408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:41.378616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:41.378823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.378823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:41.378823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.378823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.379449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:41.379740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:41.380058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.380058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:41.380058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.380058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:41.382291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:41.382468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:41.382569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:41.382705 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:41.383552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.383552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:41.383552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.383666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:41.383666 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.383666 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:41.383666 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:41.383666 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.383666 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:41.383666 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.383666 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:41.384485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.384485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:41.384485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:41.384485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:41.384485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.384485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:41.384485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:41.384485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.384485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:41.384485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:41.384485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:41.384485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.384726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:41.384920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.384920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:41.385452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:41.385730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:41.386022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:41.386022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:41.386022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:46.048730 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:46.049120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:46.049120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:46.049696 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:46.083850 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.084056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.084563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:46.084563 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.084563 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:46.084563 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:46.084563 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.084563 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:46.084563 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.084563 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:46.085298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.085298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:46.085298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:46.085298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:46.085298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.085298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:46.085298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:46.085298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.085298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:46.085298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:46.085298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:46.085298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.085747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:46.086338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:46.086743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.087232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:46.087232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:46.087232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.087232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:46.087232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.087232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:46.365920 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:46.366251 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:46.366382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:46.367537 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.367716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.367834 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.367966 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.368096 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.368227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.368336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.368446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.368584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.368691 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.368795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.368917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.369420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:46.369420 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.369420 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:46.369420 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:46.369420 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.369420 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:46.369420 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.369420 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:46.370336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.370336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:46.370336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:46.370336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:46.370336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.370336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:46.370336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:46.370336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.370336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:46.370336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:46.370336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:46.370336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.370590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:46.370832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.370832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:46.371372 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:46.371665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.371964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:46.371964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.371964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:46.371964 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:46.371964 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:46.371964 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:46.371964 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:06:46.372360 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.372522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.372900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:46.372900 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.372900 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:46.372900 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:46.372900 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.372900 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:46.372900 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.372900 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:46.373555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:46.373810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:46.373810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:46.373810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.373810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:46.373810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:46.373810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.373810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:46.373810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.373810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:46.374343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.374343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:46.374343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:46.374343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:46.374343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.374343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:46.374343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:46.374343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:06:46.374343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:46.374343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:46.374343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.374343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:46.374596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:46.374900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.374900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:46.374900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.374900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:46.375341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:46.375639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:46.375942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.375942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:46.375942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:46.375942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.375942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:46.375942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.375942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:46.376076 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:46.376172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:46.376305 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:46.376437 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:46.376693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.376849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.377249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:46.377249 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.377249 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:46.377249 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:46.377249 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.377249 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:46.377249 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.377249 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:46.377819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:46.378101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:46.378101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:46.378101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.378101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:46.378101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:46.378101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.378101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:46.378101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.378101 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:46.378672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.378672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:46.378672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:46.378672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:46.378672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.378672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:46.378672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:46.378672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:46.378672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:46.378672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.378672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:46.378988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:46.379234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.379234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:46.379234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.379234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.379610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:46.379933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:46.380275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.380275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:46.380275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.380275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:46.382572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:46.382831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:46.382957 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:46.383147 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:46.383464 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.383655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:46.383973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.384506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:46.384506 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.384506 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:46.384506 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:46.384506 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.384506 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:46.384506 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.384506 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:46.386204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.386204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:46.386204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:46.386204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:46.386204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.386204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:46.386204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:46.386204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.386204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:46.386204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:46.386204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:46.386204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.386474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:46.386700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:46.386949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:46.387291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:46.387291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:46.387291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:46.387291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:46.387291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:46.387291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:51.049267 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:51.049846 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:51.049846 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:51.050347 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:51.098437 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.098437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.098437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:51.098437 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.098437 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:51.098437 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:51.098437 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.098437 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:51.098437 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.098437 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:51.099340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.099340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:51.099340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:51.099340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:51.099340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.099340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:51.099340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:51.099340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.099340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:51.099340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:51.099340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:51.099340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.099724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:51.100123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.100123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:51.100493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:51.100993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.101556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:51.101556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.101556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:51.366476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:51.367543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:51.367768 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:51.369414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.369656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.369824 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.369971 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.370121 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.370290 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.370401 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.370527 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.370632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.370753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.370912 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.371061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.372500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:51.372500 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.372500 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:51.372500 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:51.372500 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.372500 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:51.372500 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.372500 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:51.373413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.373413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:51.373413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:51.373413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:51.373413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.373413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:51.373413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:51.373413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.373413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:51.373413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:51.373413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:51.373413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.373783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:51.373997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.373997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:51.373997 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:51.374429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:51.374792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.375177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:51.375177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.375177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:51.375177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:51.375177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:51.375177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:06:51.375329 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.375514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.376340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:51.377456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:51.377456 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.377456 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:51.377456 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:51.377456 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.377456 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:51.377456 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.377456 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:51.377988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:51.378461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:51.378461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:51.378461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.378461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:51.378461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:51.378461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.378461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:51.378461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.378461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:51.378836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.378836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:51.378836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:51.378836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:51.378836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.378836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:51.378836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:51.378836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:06:51.378836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:51.378836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:51.378836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.378836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:51.379169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:51.379491 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:51.379768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:51.380099 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:51.380373 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:51.380373 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.380373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.380713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:51.380713 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.380713 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:51.380713 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:51.380713 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.380713 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:51.380713 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.380713 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:51.381461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:51.381753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:51.381753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:51.381753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.381753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:51.381753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:51.381753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.381753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:51.381753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.381753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:51.383036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.383036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:51.383036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:51.383036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:51.383036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.383036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:51.383036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:51.383036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:51.383036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:51.383036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.383036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:51.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:51.383606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.383606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:51.383606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.383606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:51.383606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.384327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:51.384662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:51.385070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.385070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:51.385070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.385070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:51.385070 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:51.385238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:51.385421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:51.385755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.385943 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:51.386214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.387913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:51.387913 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.387913 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:51.387913 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:51.387913 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.387913 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:51.387913 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.387913 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:51.388546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.388546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:51.388546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:51.388546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:51.388546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.388546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:51.388546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:51.388546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.388546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:51.388546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:51.388546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:51.388546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.388793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:51.389013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.389013 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:51.389513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:51.389780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:51.390086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:51.390086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:51.390086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:56.051216 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:56.051216 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:56.051216 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:56.051216 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:56.105982 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.106165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:56.110211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:56.110547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.110824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:56.111092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:56.111444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:56.111444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:56.111444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.111444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:56.111444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:56.111444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.111444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:56.111444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.111444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:56.367823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:56.368300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:56.368587 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:56.369902 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.370123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.370262 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.370397 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.370545 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.370684 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.370831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.370980 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.371152 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.371303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.371441 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.371587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.372171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:56.372171 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.372171 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:56.372171 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:56.372171 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.372171 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:56.372171 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.372171 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:56.372922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.372922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:56.372922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:56.372922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:56.372922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.372922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:56.372922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:56.372922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.372922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:56.372922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:56.372922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:56.372922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.373304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:56.373693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:56.374070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.374564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:56.374564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:56.374564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.374564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:56.374564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.374564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:56.374727 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:56.374954 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:56.375126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:56.375303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:06:56.375749 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.375922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.376324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:56.376324 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.376324 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:56.376324 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:56.376324 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.376324 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:56.376324 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.376324 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:56.376911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:56.377320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:56.377320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:56.377320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.377320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:56.377320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:56.377320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.377320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:56.377320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.377320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:56.377553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.377553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:56.377553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:56.377553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:56.377553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.377553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:56.377553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:56.377553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:06:56.377553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:56.377553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:56.377553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.377553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:56.377844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:56.378207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.378207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:56.378207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.378207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:56.378375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:56.378762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:56.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:56.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:56.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:56.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.379228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:56.379696 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:56.379946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:56.380066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:56.380223 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:56.380937 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.381140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.381591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:56.381591 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.381591 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:56.381591 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:56.381591 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.381591 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:56.381591 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.381591 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:06:56.382247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:06:56.382508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:56.382508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:06:56.382508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.382508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:06:56.382508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:06:56.382508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.382508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:06:56.382508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.382508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:06:56.383122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.383122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:56.383122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:56.383122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:56.383122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.383122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:56.383122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:56.383122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:56.383122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:56.383122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.383122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:56.383541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:56.383911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:56.384331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:56.384948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:56.384948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:56.384948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:56.384948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:56.384948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.384948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:56.384948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:56.384948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.384948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:56.384948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.384948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:06:56.384948 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:06:56.385617 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:06:56.385781 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:06:56.385996 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:06:56.386426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.386626 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:06:56.386884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.387326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:06:56.387326 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.387326 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:06:56.387326 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:06:56.387326 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.387326 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:06:56.387326 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.387326 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:06:56.388096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.388096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:06:56.388096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:56.388096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:06:56.388096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.388096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:56.388096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:56.388096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.388096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:56.388096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:56.388096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:06:56.388096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.388460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:06:56.388784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.388784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:06:56.388850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:06:56.389301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:06:56.389828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:06:56.389828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:06:56.389828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:01.050154 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:01.050708 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:01.050708 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:01.051765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:01.093712 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.094221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.094598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:01.094598 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.094598 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:01.094598 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:01.094598 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.094598 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:01.094598 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.094598 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:01.095351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.095351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:01.095351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:01.095351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:01.095351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.095351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:01.095351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:01.095351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.095351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:01.095351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:01.095351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:01.095351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.095664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:01.095893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.095893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:01.095994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:01.096386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.096789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:01.096789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.096789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:01.369145 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:01.369478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:01.369593 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:01.370496 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.370647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.370738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.370849 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.370966 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.371082 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.371166 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.371281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.371366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.371693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f66bb19d-b586-45e5-a6e1-e2dbcf0bd0e2, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.371797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.371895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.372353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:01.372353 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.372353 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:01.372353 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:01.372353 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.372353 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:01.372353 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.372353 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:01.372943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.372943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:01.372943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:01.372943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:01.372943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.372943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:01.372943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:01.372943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.372943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:01.372943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:01.372943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:01.372943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.373296 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:01.373584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.373584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:01.373817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:01.374644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.375525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:01.375525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.375525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:01.375525 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:01.375525 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:01.375525 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:01.375525 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:07:01.375822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.375915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.376239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:01.376239 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.376239 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:01.376239 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:01.376239 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.376239 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:01.376239 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.376239 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:01.377881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:01.378282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:01.378561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:01.378766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:01.378766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:01.378766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:01.378766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:01.378766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:01.378766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.378766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:01.378766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:01.378766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.378766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:01.378766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.378766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:01.378945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:01.379236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:01.379556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.379556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:01.379556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:01.379556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.379556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:01.379556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.379556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:01.380378 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:01.380888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:01.380888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:01.381084 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:01.381557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.381817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.382300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:01.382300 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.382300 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:01.382300 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:01.382300 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.382300 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:01.382300 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.382300 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:01.382858 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:01.384462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:01.385279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:01.385555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:01.385555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:01.385555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:01.385555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:01.385555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.385555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:01.385555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:01.385555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.385555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:01.385555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.385555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:01.385555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:01.385792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.386211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:01.386685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:01.386685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:01.386685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:01.386685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:01.386685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.386685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:01.386806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.387313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:01.387313 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.387313 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:01.387313 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:01.387313 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.387313 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:01.387313 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.387313 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:01.387791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.387791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:01.387791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:01.387791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:01.387791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.387791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:01.387791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:01.387791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.387791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:01.387791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:01.387791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:01.387791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.388307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:01.388765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:01.390249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:01.391115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:01.391115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:01.391115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:01.391115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:01.391115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:01.391115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:06.050607 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:06.052757 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:06.052757 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:06.052757 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:06.088950 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.089175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:06.091481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:06.091864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.092108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:06.092449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:06.092812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:06.092812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:06.092812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.092812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:06.092812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:06.092812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.092812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:06.092812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.092812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:06.370951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:06.370951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:06.370951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:06.371169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.371369 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.371528 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.371677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.371816 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.371963 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.372125 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.372282 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.372429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.372574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.372713 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.372857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.373447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:06.373447 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.373447 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:06.373447 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:06.373447 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.373447 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:06.373447 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.373447 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:06.374645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.374645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:06.374645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:06.374645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:06.374645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.374645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:06.374645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:06.374645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.374645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:06.374645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:06.374645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:06.374645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.374919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:06.375167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.375167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:06.376530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:06.376824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.377279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:06.377279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.377279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:06.377768 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:06.378039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:06.378683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:06.378683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:07:06.378933 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.379148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.379624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:06.379624 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.379624 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:06.379624 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:06.379624 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.379624 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:06.379624 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.379624 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:06.380628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:06.380968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:06.380968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:06.380968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.380968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:06.380968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:06.380968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.380968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:06.380968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.380968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:06.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:06.382623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.382623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:06.382623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:06.382623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:06.382623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.382623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:06.382623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:06.382623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:07:06.382623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:06.382623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:06.382623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.382623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:06.382947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:06.383311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.383311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:06.383311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.383311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:06.385218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:06.385585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:06.385952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.385952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:06.385952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:06.385952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.385952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:06.385952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.385952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:06.385952 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:06.385952 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:06.385952 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:06.386162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:06.387039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.387132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.388264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:06.388264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.388264 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:06.388264 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:06.388264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.388264 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:06.388264 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.388264 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:06.388919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:06.389214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:06.389597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:06.391176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:06.391176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:06.391176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:06.391176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:06.391176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.391176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:06.391176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:06.391176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.391176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:06.391176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.391176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:06.391176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:06.391479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.391767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:06.392100 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:06.392100 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:06.392100 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:06.392100 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.392100 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:06.392100 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.392233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:06.392233 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.392233 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:06.392233 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:06.392233 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.392233 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:06.392233 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.392233 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:06.393173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.393173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:06.393173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:06.393173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:06.393173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.393173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:06.393173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:06.393173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.393173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:06.393173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:06.393173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:06.393173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.393887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:06.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:06.395988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:06.396675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:06.396675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:06.396675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:06.396675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:06.396675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:06.396675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:11.051246 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:11.051760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:11.051818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:11.052494 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:11.103884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.104043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:11.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:11.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:11.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:11.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:11.105234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.105234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:11.105234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:11.105234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:11.105234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.105234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:11.105234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:11.105234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.105234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:11.105234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:11.105234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:11.105234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:11.105774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.105774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:11.106722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:11.107095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.107509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:11.107509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.107509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:11.372429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:11.372697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:11.372815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:11.374027 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.374255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.374389 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.374549 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.374674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.374810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.374894 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.375014 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.375138 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.375257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.375357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.375453 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.375551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.376103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:11.376103 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.376103 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:11.376103 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:11.376103 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.376103 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:11.376103 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.376103 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:11.377474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.377474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:11.377474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:11.377474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:11.377474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.377474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:11.377474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:11.377474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.377474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:11.377474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:11.377474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:11.377474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:11.378040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.378040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:11.378462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:11.378764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.379166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:11.379166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.379166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:11.379825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:11.380130 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:11.380345 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:11.380528 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:07:11.380935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.381141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.381558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:11.381558 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.381558 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:11.381558 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:11.381558 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.381558 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:11.381558 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.381558 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:11.382376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:11.382650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:11.382650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:11.382650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.382650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:11.382650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:11.382650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.382650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:11.382650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.382650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:11.383200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.383200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:11.383200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:11.383200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:11.383200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.383200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:11.383200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:11.383200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:07:11.383200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:11.383200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:11.383200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.383200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:11.383546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:11.383883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:11.384324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:11.384790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:11.384790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:11.384790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:11.384790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:11.384790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:11.384790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.384790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:11.384790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:11.384790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.384790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:11.384790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.384790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:11.385019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:11.385577 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:11.385870 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:11.386118 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:11.386786 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.386992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.387476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:11.387476 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.387476 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:11.387476 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:11.387476 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.387476 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:11.387476 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.387476 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:11.388795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:11.389322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:11.389322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:11.389322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.389322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:11.389322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:11.389322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.389322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:11.389322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.389322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:11.390478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.390478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:11.390478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:11.390478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:11.390478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.390478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:11.390478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:11.390478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:11.390478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:11.390478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.390478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:11.390702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:11.390965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.390965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:11.390965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.390965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:11.391564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.392599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:11.392933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:11.393253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.393253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:11.393253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.393253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:11.393253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:11.394014 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:11.394245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:11.394542 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.394725 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:11.394985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.395550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:11.395550 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.395550 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:11.395550 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:11.395550 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.395550 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:11.395550 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.395550 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:11.396477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.396477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:11.396477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:11.396477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:11.396477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.396477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:11.396477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:11.396477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.396477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:11.396477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:11.396477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:11.396477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.396721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:11.396954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.396954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:11.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:11.398443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:11.398739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:11.398739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:11.398739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:16.051804 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:16.052250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:16.052380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:16.053164 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:16.098800 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.099445 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.102628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:16.102628 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.102628 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:16.102628 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:16.102628 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.102628 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:16.102628 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.102628 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:16.103876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.103876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:16.103876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:16.103876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:16.103876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.103876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:16.103876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:16.103876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.103876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:16.103876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:16.103876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:16.103876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.104111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:16.104340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.104340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:16.105475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:16.105749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.106045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:16.106045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.106045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:16.372874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:16.373422 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:16.373582 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:16.375104 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.375695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.375695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.375695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.375812 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.375984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.376176 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.376390 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.376543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.376681 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.376815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.376948 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.377113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.377757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:16.377757 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.377757 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:16.377757 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:16.377757 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.377757 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:16.377757 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.377757 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:16.378904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.378904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:16.378904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:16.378904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:16.378904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.378904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:16.378904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:16.378904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.378904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:16.378904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:16.378904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:16.378904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.379381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:16.379831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.379831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:16.380103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:16.380587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.381087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:16.381087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.381087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:16.381087 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:16.381087 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:16.381544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:16.381544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:07:16.381759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.381987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.382587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:16.382587 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.382587 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:16.382587 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:16.382587 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.382587 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:16.382587 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.382587 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:16.383130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:16.383610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:16.383610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:16.383610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.383610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:16.383610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:16.383610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.383610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:16.383610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.383610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:16.383839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.383839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:16.383839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:16.383839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:16.383839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.383839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:16.383839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:16.383839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:07:16.383839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:16.383839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:16.383839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.383839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:16.384151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:16.384656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:16.385097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:16.385636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:16.385636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:16.385636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:16.385636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:16.385636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:16.385636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.385636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:16.385636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:16.385636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.385636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:16.385636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.385636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:16.385966 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:16.385966 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:16.386037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:16.386153 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:16.386473 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.386669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.387207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:16.387207 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.387207 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:16.387207 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:16.387207 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.387207 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:16.387207 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.387207 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:16.387688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:16.388175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:16.388175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:16.388175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.388175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:16.388175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:16.388175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.388175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:16.388175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.388175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:16.388472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.388472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:16.388472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:16.388472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:16.388472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.388472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:16.388472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:16.388472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:16.388472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:16.388472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.388472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:16.388837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:16.389293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:16.389738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:16.390306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:16.390306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:16.390306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:16.390306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:16.390306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.390306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:16.390306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:16.390306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.390306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:16.390306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.390306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:16.393146 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:16.393421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:16.393568 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:16.393697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:16.394018 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.394249 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:16.394505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.394970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:16.394970 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.394970 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:16.394970 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:16.394970 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.394970 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:16.394970 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.394970 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:16.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:16.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:16.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:16.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:16.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:16.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:16.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:16.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:16.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.396161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:16.396677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.396677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:16.396877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:16.397386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:16.397922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:16.397922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:16.397922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:21.053265 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:21.054822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:21.054935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:21.055611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:21.071263 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=1070,neutron:UPDATE=2,neutron:DELETE=10 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:07:21.098432 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.098632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.099155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:21.099155 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.099155 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:21.099155 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:21.099155 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.099155 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:21.099155 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.099155 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:21.099956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.099956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:21.099956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:21.099956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:21.099956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.099956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:21.099956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:21.099956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.099956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:21.099956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:21.099956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:21.099956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.100393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:21.100919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.100919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:21.101070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:21.101574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.102258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:21.102258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.102258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:21.373415 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:21.374597 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:21.374597 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:21.375882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.376082 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.376227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.376362 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.376470 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.376568 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.376670 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.376769 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.376867 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.376961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.377063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.377159 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.377301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.377813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:21.377813 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.377813 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:21.377813 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:21.377813 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.377813 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:21.377813 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.377813 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:21.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:21.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:21.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:21.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:21.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:21.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:21.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:21.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:21.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.379078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:21.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.379599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:21.379778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:21.380179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.380640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:21.380640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.380640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:21.380776 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:21.381022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:21.381131 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:21.381264 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:07:21.381562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.381721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.382122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:21.382122 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.382122 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:21.382122 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:21.382122 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.382122 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:21.382122 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.382122 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:21.382683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:21.383165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:21.383165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:21.383165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.383165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:21.383165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:21.383165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.383165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:21.383165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.383165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:21.383493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.383493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:21.383493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:21.383493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:21.383493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.383493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:21.383493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:21.383493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:07:21.383493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:21.383493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:21.383493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.383493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:21.383726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:21.384016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:21.384325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:21.384888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:21.384888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:21.384888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:21.384888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:21.384888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:21.384888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.384888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:21.384888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:21.384888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.384888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:21.384888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.384888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:21.386019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:21.386278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:21.386366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:21.386465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:21.386765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.386941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.387396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:21.387396 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.387396 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:21.387396 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:21.387396 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.387396 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:21.387396 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.387396 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:21.387940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:21.388414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:21.388414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:21.388414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.388414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:21.388414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:21.388414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.388414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:21.388414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.388414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:21.388688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.388688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:21.388688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:21.388688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:21.388688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.388688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:21.388688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:21.388688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:21.388688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:21.388688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.388688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:21.389032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:21.389533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:21.389987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:21.390590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:21.390590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:21.390590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:21.390590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:21.390590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.390590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:21.390590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:21.390590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.390590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:21.390590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.390590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:21.393465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:21.393794 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:21.393938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:21.394203 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:21.394513 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.394824 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:21.395117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.395737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:21.395737 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.395737 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:21.395737 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:21.395737 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.395737 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:21.395737 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.395737 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:21.396780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.396780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:21.396780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:21.396780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:21.396780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.396780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:21.396780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:21.396780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.396780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:21.396780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:21.396780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:21.396780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.397130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:21.397478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.397478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:21.397901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:21.398398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:21.398889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:21.398889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:21.398889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:26.054875 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:26.055265 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:26.055299 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:26.056005 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:26.106267 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.106513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.109416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:26.109416 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.109416 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:26.109416 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:26.109416 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.109416 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:26.109416 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.109416 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:26.110104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.110104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:26.110104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:26.110104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:26.110104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.110104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:26.110104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:26.110104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.110104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:26.110104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:26.110104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:26.110104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.110516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:26.110876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:26.111320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.111836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:26.111836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:26.111836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.111836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:26.111836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.111836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:26.374499 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:26.374845 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:26.375074 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:26.377635 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.377859 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.378014 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.378183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.378364 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.378519 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.378689 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.378818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.378955 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.379084 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.379246 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.379334 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.379498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.379939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:26.379939 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.379939 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:26.379939 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:26.379939 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.379939 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:26.379939 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.379939 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:26.382255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.382255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:26.382255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:26.382255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:26.382255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.382255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:26.382255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:26.382255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.382255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:26.382255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:26.382255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:26.382255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.382616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:26.382879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.382879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:26.382879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:26.386210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:26.386788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.387337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:26.387337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.387337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:26.387337 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:26.387337 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:26.387337 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:07:26.387337 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.387337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.387549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:26.387549 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.387549 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:26.387549 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:26.387549 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.387549 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:26.387549 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.387549 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:26.388089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:26.388595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:26.388595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:26.388595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.388595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:26.388595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:26.388595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.388595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:26.388595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.388595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:26.388595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:26.388994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.388994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:26.388994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:26.388994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:26.388994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.388994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:26.388994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:26.388994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:07:26.388994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:26.388994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:26.388994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.388994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:26.389503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:26.389854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.389854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:26.389854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.389854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:26.389968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:26.390635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:26.391667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.391667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:26.391667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:26.391667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.391667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:26.391667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.391667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:26.391667 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:26.391667 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:26.391667 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:26.392467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.393282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:26.393892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.393892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:26.393892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.393892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:26.393892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:26.393892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:26.393892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.393892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:26.393892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:26.393892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:26.393892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:26.393892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.394248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:26.394625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:26.395076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:26.396411 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:26.396861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:26.396861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:26.396861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.396861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:26.396861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.396861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:26.396861 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.396861 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:26.396861 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:26.397152 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.397152 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:26.397152 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.397152 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:26.397730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.397730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:26.397730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:26.397730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:26.397730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.397730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:26.397730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:26.397730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.397730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:26.397730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:26.397730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:26.397730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.398173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:26.398386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:26.398626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:26.398899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:26.398899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:26.398899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:26.398899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:26.398899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:26.398899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:31.056322 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:31.057564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:31.057670 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:31.058334 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:31.112986 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.112986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.113436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:31.113436 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.113436 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:31.113436 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:31.113436 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.113436 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:31.113436 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.113436 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:31.113812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.113812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:31.113812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:31.113812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:31.113812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.113812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:31.113812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:31.113812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.113812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:31.113812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:31.113812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:31.113812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.114181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:31.114524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.114524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:31.114592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:31.115632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.116589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:31.116589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.116589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:31.375013 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:31.375400 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:31.375525 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:31.378427 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.378631 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.378806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.378977 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.379140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.379317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.379546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.379597 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.379727 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.379859 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.380000 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.380157 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.380324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.382512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:31.382512 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.382512 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:31.382512 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:31.382512 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.382512 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:31.382512 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.382512 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:31.384633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.384633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:31.384633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:31.384633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:31.384633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.384633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:31.384633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:31.384633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.384633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:31.384633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:31.384633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:31.384633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.384992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:31.385319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.385319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:31.385319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:31.387281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:31.387634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.387936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:31.387936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.387936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:31.388706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:31.388864 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:31.389073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:07:31.389935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.390160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.390594 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:31.391113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:31.391113 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.391113 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:31.391113 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:31.391113 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.391113 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:31.391113 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.391113 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:31.392949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:31.393365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:31.393365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:31.393365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.393365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:31.393365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:31.393365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.393365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:31.393365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.393365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:31.394043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.394043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:31.394043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:31.394043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:31.394043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.394043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:31.394043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:31.394043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:07:31.394043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:31.394043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:31.394043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.394043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:31.394661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:31.394904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:31.395199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:31.395536 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:31.395805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:31.395805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:31.395805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.395805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.395805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:31.395805 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.395805 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:31.395805 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:31.395805 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.395805 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:31.395805 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.395805 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:31.396646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:31.397295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:31.397531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:31.397728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:31.397728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:31.397728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:31.397728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:31.397728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.397728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:31.397728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:31.397728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.397728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:31.397728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.397728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:31.397728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:31.398239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.399490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:31.399832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:31.399832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:31.399832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:31.399832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.399832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:31.399832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:31.399955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:31.400287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.400518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:31.400763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:31.401090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:31.401090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:31.401090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:31.401090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:31.401090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:31.401090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:31.401090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:31.401090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:31.401090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:31.401090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:36.057653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:36.060022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:36.060022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:36.060022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:36.092890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.093139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:36.094948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:36.095328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.095597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:36.095906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:36.096287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:36.096287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:36.096287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.096287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:36.096287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:36.096287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.096287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:36.096287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.096287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:36.375506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:36.376655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:36.376655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:36.377174 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.377377 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.377524 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.377661 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.377805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.377944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.378082 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.378242 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.378382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.378542 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.378681 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.378821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.378962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:36.381953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:36.382460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.382836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:36.383269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:36.384107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:36.384107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:36.384107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:36.384107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.384107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:36.384107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:36.384107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.384107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:36.384107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.384107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:36.389276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:36.389276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:36.389276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:36.389276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:07:36.389571 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.389694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.391943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:36.392638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:36.393104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.393479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:36.393874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:36.394330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:36.394876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:36.394876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:36.394876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.394876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:36.394876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:36.394876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.394876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:36.394876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.394876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:36.396017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:36.396017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:36.396017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:36.396017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:36.397294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.397566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.399031 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.401853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:36.402237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:36.402509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:36.402954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:36.403344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:36.403747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:36.403747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.403747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:36.403747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:36.403747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.403747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:36.403747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.403747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:36.403747 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:36.403747 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:36.403747 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:36.403940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.403940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:36.403940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.405265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:36.405265 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.405265 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:36.405265 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:36.405265 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.405265 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:36.405265 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.405265 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:36.406135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.406135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:36.406135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:36.406135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:36.406135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.406135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:36.406135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:36.406135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.406135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:36.406135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:36.406135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:36.406135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:36.406837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.406837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:36.407140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:36.407715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:36.408293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:36.408293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:36.408293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:41.058459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:41.059091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:41.059091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:41.059539 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:41.093041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.093262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.093751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:41.093751 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.093751 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:41.093751 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:41.093751 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.093751 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:41.093751 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.093751 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:41.094512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.094512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:41.094512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:41.094512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:41.094512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.094512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:41.094512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:41.094512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.094512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:41.094512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:41.094512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:41.094512 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.094930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:41.095389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.095389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:41.095726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:41.096238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.096568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:41.096568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.096568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:41.376013 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:41.376405 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:41.376478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:41.377515 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.377643 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.377752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.377852 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.378004 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.378077 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.378157 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.378285 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.378374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.378465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.378556 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.378648 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.378741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.379398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:41.379398 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.379398 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:41.379398 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:41.379398 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.379398 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:41.379398 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.379398 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:41.380097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.380097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:41.380097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:41.380097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:41.380097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.380097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:41.380097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:41.380097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.380097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:41.380097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:41.380097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:41.380097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.380530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:41.380902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:41.381400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.382003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:41.382003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:41.382003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.382003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:41.382003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.382003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:41.389147 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:41.389444 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:41.389529 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:41.389647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:07:41.389907 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.390058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.390432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:41.390432 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.390432 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:41.390432 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:41.390432 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.390432 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:41.390432 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.390432 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:41.390936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:41.391487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:41.391938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:41.392366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:41.392366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:41.392366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:41.392366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:41.392366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:41.392366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.392366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:41.392366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:41.392366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.392366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:41.392366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.392366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:41.392756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:41.393444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:41.394017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.394017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:41.394017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:41.394017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.394017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:41.394017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.394017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:41.395931 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:41.396141 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:41.396218 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:41.396325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:41.396533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.396692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.397017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:41.397017 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.397017 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:41.397017 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:41.397017 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.397017 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:41.397017 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.397017 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:41.397520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:41.397976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:41.397976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:41.397976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.397976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:41.397976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:41.397976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.397976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:41.397976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.397976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:41.398392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.398392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:41.398392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:41.398392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:41.398392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.398392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:41.398392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:41.398392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:41.398392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:41.398392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.398392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:41.398767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:41.399230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:41.399658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:41.400252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:41.400252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:41.400252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:41.400252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:41.400252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.400252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:41.400252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:41.400252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.400252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:41.400252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.400252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:41.401756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:41.401953 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:41.402028 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:41.402161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:41.402390 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.402553 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:41.402800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.403102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:41.403102 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.403102 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:41.403102 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:41.403102 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.403102 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:41.403102 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.403102 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:41.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:41.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:41.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:41.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:41.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:41.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:41.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:41.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:41.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.404006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:41.404430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:41.404890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:41.405504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:41.405504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:41.405504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:41.405504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:41.405504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:41.405504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:46.058934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:46.059380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:46.059380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:46.059958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:46.094986 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.095164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.095644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:46.095644 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.095644 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:46.095644 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:46.095644 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.095644 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:46.095644 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.095644 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:46.096290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.096290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:46.096290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:46.096290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:46.096290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.096290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:46.096290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:46.096290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.096290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:46.096290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:46.096290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:46.096290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.096678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:46.096981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:46.097416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:46.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:46.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:46.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:46.376561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:46.376857 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:46.376947 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:46.377938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.378092 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.378201 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.378306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.378395 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.378493 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.378575 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.378662 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.378752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.380667 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.380779 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.380879 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.380974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.381389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:46.381389 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.381389 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:46.381389 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:46.381389 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.381389 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:46.381389 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.381389 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:46.381984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.381984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:46.381984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:46.381984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:46.381984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.381984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:46.381984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:46.381984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.381984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:46.381984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:46.381984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:46.381984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.382284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:46.382507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.382507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:46.382697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:46.383047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.383419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:46.383419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.383419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:46.391401 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:46.391401 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:46.391401 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:46.391595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:07:46.391883 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.392095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.392670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:46.392670 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.392670 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:46.392670 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:46.392670 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.392670 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:46.392670 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.392670 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:46.393431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:46.393729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:46.393729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:46.393729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.393729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:46.393729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:46.393729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.393729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:46.393729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.393729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:46.394721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.394721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:46.394721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:46.394721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:46.394721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.394721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:46.394721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:46.394721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:07:46.394721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:46.394721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:46.394721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.394721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:46.394961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:46.395291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.395291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:46.395291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.395291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:46.395877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:46.396169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:46.396502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.396502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:46.396502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:46.396502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.396502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:46.396502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.396502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:46.396502 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:46.396502 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:46.396680 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:46.396680 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:46.397059 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.397360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.397938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:46.397938 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.397938 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:46.397938 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:46.397938 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.397938 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:46.397938 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.397938 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:46.398607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:46.398873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:46.398873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:46.398873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.398873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:46.398873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:46.398873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.398873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:46.398873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.398873 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:46.400653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.400653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:46.400653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:46.400653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:46.400653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.400653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:46.400653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:46.400653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:46.400653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:46.400653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.400653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:46.400886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:46.401136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.401136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:46.401136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.401136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.402054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:46.402371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:46.402681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.402681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:46.402681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.402681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:46.403011 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:46.403737 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:46.403926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:46.404046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:46.404538 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.404805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:46.405206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.405770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:46.405770 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.405770 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:46.405770 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:46.405770 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.405770 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:46.405770 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.405770 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:46.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:46.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:46.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:46.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:46.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:46.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:46.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:46.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:46.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.406968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:46.407226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.407226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:46.410077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:46.410544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:46.411051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:46.411051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:46.411051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:51.059390 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:51.059925 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:51.059925 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:51.060783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:51.091711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.091901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.092437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:51.092437 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.092437 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:51.092437 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:51.092437 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.092437 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:51.092437 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.092437 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:51.093245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.093245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:51.093245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:51.093245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:51.093245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.093245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:51.093245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:51.093245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.093245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:51.093245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:51.093245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:51.093245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.093639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:51.094046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.094046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:51.094321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:51.094623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.094943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:51.094943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.094943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:51.377029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:51.377220 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:51.377296 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:51.378319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.378501 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.378667 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.378825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.378965 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.379127 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.379291 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.379448 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.379592 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.379736 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.379875 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.380012 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.380203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.380669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:51.380669 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.380669 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:51.380669 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:51.380669 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.380669 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:51.380669 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.380669 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:51.381291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.381291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:51.381291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:51.381291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:51.381291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.381291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:51.381291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:51.381291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.381291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:51.381291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:51.381291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:51.381291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.381663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:51.381919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.381919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:51.382086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:51.382440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.382823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:51.382823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.382823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:51.391312 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:51.391721 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:51.391847 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:51.391973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:07:51.392402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.392587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.393010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:51.393010 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.393010 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:51.393010 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:51.393010 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.393010 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:51.393010 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.393010 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:51.393785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:51.394119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:51.394119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:51.394119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.394119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:51.394119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:51.394119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.394119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:51.394119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.394119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:51.394636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.394636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:51.394636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:51.394636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:51.394636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.394636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:51.394636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:51.394636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:07:51.394636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:51.394636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:51.394636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.394636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:51.394874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:51.395121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.395121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:51.395121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.395121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:51.395659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:51.395945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:51.396254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.396254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:51.396254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:51.396254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.396254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:51.396254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.396254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:51.396572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:51.396804 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:51.396919 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:51.397052 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:51.397381 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.397561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.397906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:51.397906 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.397906 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:51.397906 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:51.397906 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.397906 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:51.397906 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.397906 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:51.398630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:51.398921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:51.398921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:51.398921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.398921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:51.398921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:51.398921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.398921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:51.398921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.398921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:51.399511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.399511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:51.399511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:51.399511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:51.399511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.399511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:51.399511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:51.399511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:51.399511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:51.399511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.399511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:51.399739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:51.399985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.399985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:51.399985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.399985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:51.400971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:51.401310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.401310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:51.401310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.401310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:51.403740 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:51.403982 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:51.404109 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:51.404306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:51.404564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.404739 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:51.404989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.405384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:51.405384 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.405384 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:51.405384 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:51.405384 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.405384 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:51.405384 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.405384 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:51.406176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.406176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:51.406176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:51.406176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:51.406176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.406176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:51.406176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:51.406176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.406176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:51.406176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:51.406176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:51.406176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.406429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:51.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:51.407170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:51.407844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:51.407844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:51.407844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:56.059735 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:56.060311 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:56.060311 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:56.060853 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:56.094520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.094780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.095285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:56.095285 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.095285 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:56.095285 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:56.095285 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.095285 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:56.095285 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.095285 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:56.096201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.096201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:56.096201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:56.096201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:56.096201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.096201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:56.096201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:56.096201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.096201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:56.096201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:56.096201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:56.096201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.097155 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:56.097906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:56.098733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.099612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:56.099612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:56.099612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.099612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:56.099612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.099612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:56.377333 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:56.377677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:56.377826 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:56.378934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.379119 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.379249 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.379351 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.379478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.379571 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.379663 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.379753 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.379837 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.379929 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.380025 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.380137 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.380246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.381397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:56.381397 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.381397 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:56.381397 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:56.381397 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.381397 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:56.381397 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.381397 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:56.382084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.382084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:56.382084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:56.382084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:56.382084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.382084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:56.382084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:56.382084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.382084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:56.382084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:56.382084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:56.382084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:56.383397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:56.383816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.384139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:56.384139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:56.384139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.384139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:56.384139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.384139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:56.391764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:56.392053 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:56.392210 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:56.392318 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:07:56.392764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.392926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.396547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:56.396893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:56.397244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:56.397670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:56.397902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:56.398228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:56.398228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:56.398228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.398228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:56.398228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:56.398228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.398228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:56.398228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.398228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:56.398228 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:56.399565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:56.399565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:56.399565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:56.400084 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.400387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.402715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:07:56.403022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:56.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.403558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:56.403766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:56.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:56.404334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:56.404334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.404334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:56.404334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:56.404334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.404334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:56.404334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.404334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:07:56.404334 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:07:56.405668 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:07:56.405668 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:07:56.405668 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:07:56.405949 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.406138 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:07:56.406443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:56.408242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:07:56.409142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.409851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:07:56.410533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:07:56.411024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:07:56.411024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:07:56.411024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:07:56.411024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:07:56.411024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:07:56.411024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:07:56.411024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:07:56.411024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:07:56.411024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:07:56.411024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:01.060114 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:01.060622 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:01.060622 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:01.060989 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:01.090862 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.091133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.091810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:01.091810 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.091810 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:01.091810 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:01.091810 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.091810 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:01.091810 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.091810 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:01.092536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.092536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:01.092536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:01.092536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:01.092536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.092536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:01.092536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:01.092536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.092536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:01.092536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:01.092536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:01.092536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.092785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:01.093027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.093027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:01.093202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:01.093519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.093955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:01.093955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.093955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:01.377837 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:01.378013 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:01.378132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:01.379117 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.379251 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.379365 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.379465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.379587 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.379745 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.379810 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.379904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.380000 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.380124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.380235 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.380330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.380427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.381025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:01.381025 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.381025 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:01.381025 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:01.381025 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.381025 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:01.381025 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.381025 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:01.381899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.381899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:01.381899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:01.381899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:01.381899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.381899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:01.381899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:01.381899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.381899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:01.381899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:01.381899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:01.381899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.382334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:01.382699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:01.383229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.383853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:01.383853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:01.383853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.383853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:01.383853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.383853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:01.392175 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:01.392491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:01.392571 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:01.392670 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:08:01.393022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.393232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.393647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:01.393647 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.393647 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:01.393647 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:01.393647 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.393647 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:01.393647 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.393647 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:01.394206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:01.394638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:01.394638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:01.394638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.394638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:01.394638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:01.394638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.394638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:01.394638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.394638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:01.394881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.394881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:01.394881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:01.394881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:01.394881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.394881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:01.394881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:01.394881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:08:01.394881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:01.394881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:01.394881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.394881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:01.395226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:01.395558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:01.395899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:01.396405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:01.396405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:01.396405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:01.396405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:01.396405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:01.396405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.396405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:01.396405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:01.396405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.396405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:01.396405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.396405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:01.399399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:01.399638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:01.399727 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:01.399826 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:01.400117 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.400346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.400731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:01.400731 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.400731 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:01.400731 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:01.400731 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.400731 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:01.400731 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.400731 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:01.401414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:01.401707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:01.401707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:01.401707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.401707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:01.401707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:01.401707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.401707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:01.401707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.401707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:01.401948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.401948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:01.401948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:01.401948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:01.401948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.401948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:01.401948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:01.401948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:01.401948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:01.401948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.401948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:01.402163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:01.402401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.402401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:01.402401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.402401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.402672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:01.402970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:01.403283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.403283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:01.403283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.403283 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:01.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:01.405717 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:01.405814 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:01.405938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:01.406256 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.406465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:01.406746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.407179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:01.407179 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.407179 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:01.407179 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:01.407179 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.407179 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:01.407179 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.407179 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:01.407925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.407925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:01.407925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:01.407925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:01.407925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.407925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:01.407925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:01.407925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.407925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:01.407925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:01.407925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:01.407925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.408235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:01.408433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.408433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:01.408627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:01.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:01.409311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:01.409311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:01.409311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:06.060560 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:06.061028 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:06.061064 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:06.061638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:06.094431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.094616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.095043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:06.095043 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.095043 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:06.095043 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:06.095043 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.095043 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:06.095043 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.095043 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:06.095676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.095676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:06.095676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:06.095676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:06.095676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.095676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:06.095676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:06.095676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.095676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:06.095676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:06.095676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:06.095676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.095986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:06.096679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.096679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:06.097783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:06.098254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.098706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:06.098706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.098706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:06.378183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:06.379123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:06.379123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:06.379339 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.379498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.379614 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.379720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.379815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.379908 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.380001 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.381561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.381561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.381561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.381561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.381561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.381561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.381740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:06.381740 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.381740 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:06.381740 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:06.381740 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.381740 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:06.381740 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.381740 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:06.381879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.381879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:06.381879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:06.381879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:06.381879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.381879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:06.381879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:06.381879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.381879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:06.381879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:06.381879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:06.381879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.382062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:06.382285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.382285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:06.382481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:06.382743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.383050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:06.383050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.383050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:06.392427 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:06.392626 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:06.392716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:06.392802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:08:06.393029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.393178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.393502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:06.393502 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.393502 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:06.393502 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:06.393502 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.393502 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:06.393502 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.393502 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:06.393970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:06.394261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:06.394261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:06.394261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.394261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:06.394261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:06.394261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.394261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:06.394261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.394261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:06.394509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.394509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:06.394509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:06.394509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:06.394509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.394509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:06.394509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:06.394509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:08:06.394509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:06.394509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:06.394509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.394509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:06.394713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:06.394937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.394937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:06.394937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.394937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:06.395132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:06.395396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:06.395676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.395676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:06.395676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:06.395676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.395676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:06.395676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.395676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:06.399622 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:06.399801 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:06.399889 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:06.399978 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:06.400214 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.400378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.400711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:06.400711 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.400711 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:06.400711 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:06.400711 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.400711 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:06.400711 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.400711 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:06.401173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:06.401433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:06.401433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:06.401433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.401433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:06.401433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:06.401433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.401433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:06.401433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.401433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:06.402482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.402482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:06.402482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:06.402482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:06.402482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.402482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:06.402482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:06.402482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:06.402482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:06.402482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.402482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:06.402717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:06.402971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:06.403232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:06.403540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:06.403540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:06.403540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:06.403540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:06.403540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.403540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:06.403540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:06.403540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.403540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:06.403540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.403540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:06.405738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:06.405883 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:06.405966 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:06.406076 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:06.406317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.406530 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:06.406852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.407238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:06.407238 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.407238 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:06.407238 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:06.407238 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.407238 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:06.407238 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.407238 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:06.407752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.407752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:06.407752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:06.407752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:06.407752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.407752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:06.407752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:06.407752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.407752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:06.407752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:06.407752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:06.407752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.407938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:06.408163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.408163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:06.408402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:06.408648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:06.408978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:06.408978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:06.408978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:11.061183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:11.061730 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:11.061730 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:11.062421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:11.115211 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.115472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.115959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:11.115959 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.115959 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:11.115959 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:11.115959 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.115959 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:11.115959 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.115959 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:11.116691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.116691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:11.116691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:11.116691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:11.116691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.116691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:11.116691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:11.116691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.116691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:11.116691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:11.116691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:11.116691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.117036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:11.117419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:11.117792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.118304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:11.118304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:11.118304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.118304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:11.118304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.118304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:11.378533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:11.378895 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:11.379016 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:11.380122 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.380321 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.380465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.380604 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.380757 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.380898 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.381050 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.381218 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.381361 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.381504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.381645 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.381785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-768a046c-1947-42e8-9b1f-314c1b73affc, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.381924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.382535 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:11.383107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.383107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:11.383107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:11.383107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:11.383107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.383107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:11.383107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:11.383107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.383107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:11.383107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:11.383107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:11.383107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.383391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:11.383605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.383605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:11.383941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:11.384295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.384623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:11.384623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.384623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:11.393100 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:11.393100 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:11.393182 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:11.393326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:08:11.393839 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.394065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.395779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:11.396530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:11.396835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.397046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:11.397284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:11.397535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:11.397839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:11.397839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:11.397839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.397839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:11.397839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:11.397839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.397839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:11.397839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.397839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:11.399887 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:11.400043 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:11.400163 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:11.400373 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:11.400674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.400908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:11.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:11.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:11.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:11.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:11.402112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:11.402399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:11.402399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:11.402399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.402399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:11.402399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:11.402399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.402399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:11.402399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.402399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:11.402839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.402839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:11.402839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:11.402839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:11.402839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.402839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:11.402839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:11.402839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:11.402839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:11.402839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.402839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:11.403551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:11.403810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:11.404036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:11.404378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:11.404378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:11.404378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:11.404378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:11.404378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.404378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:11.404378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:11.404378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.404378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:11.404378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.404378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:11.405963 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:11.406207 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:11.406298 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:11.406427 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:11.410212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.410212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:11.410212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.410212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:11.410212 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.410212 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:11.410212 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:11.410212 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.410212 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:11.410212 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.410212 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:11.410212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:11.410485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:11.410729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:11.410729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:11.410729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:11.410729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:11.410729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:11.410729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.410729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:11.410729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:11.410729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.410729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:11.410729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.410729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:11.410926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:11.411219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:11.411513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:11.411513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:11.411513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:16.061698 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:16.062284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:16.062284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:16.062800 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:16.101570 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.101855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.102566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:16.102566 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.102566 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:16.102566 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:16.102566 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.102566 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:16.102566 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.102566 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:16.103291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.103291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:16.103291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:16.103291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:16.103291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.103291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:16.103291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:16.103291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.103291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:16.103291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:16.103291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:16.103291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.103646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:16.103986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.103986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:16.104049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:16.104504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.104969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:16.104969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.104969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:16.378955 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:16.379393 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:16.379506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:16.381494 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.381625 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.381729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fb18b94-7a38-4aab-9569-672e24d0c07f, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.381829 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-27b7a4c4-a577-4c4c-8282-939f58a97615, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.381920 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.382018 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.382117 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.382233 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.382331 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.382423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.382513 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-6de85ce9-38ca-4cad-b472-ee3299d66135, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.382606 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:16.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:16.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:16.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:16.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.383335 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:16.383961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.383961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:16.383961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:16.383961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:16.383961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.383961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:16.383961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:16.383961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.383961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:16.383961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:16.383961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:16.383961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.384265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:16.384509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.384509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:16.385051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:16.385392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.385686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:16.385686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.385686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:16.393177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:16.393599 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:16.393729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:16.393870 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:08:16.394220 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.394698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.395137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:16.395137 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.395137 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:16.395137 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:16.395137 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.395137 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:16.395137 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.395137 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:16.395917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:16.396248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:16.396248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:16.396248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.396248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:16.396248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:16.396248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.396248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:16.396248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.396248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:16.396827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.396827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:16.396827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:16.396827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:16.396827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.396827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:16.396827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:16.396827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:08:16.396827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:16.396827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:16.396827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.396827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:16.397040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:16.397312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.397312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:16.397312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.397312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:16.397925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:16.398240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:16.398563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.398563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:16.398563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:16.398563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.398563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:16.398563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.398563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:16.400300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:16.400630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:16.400749 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:16.400912 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:16.401206 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.401415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.401776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:16.401776 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.401776 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:16.401776 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:16.401776 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.401776 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:16.401776 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.401776 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:16.402336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:16.402668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:16.402668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:16.402668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.402668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:16.402668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:16.402668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.402668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:16.402668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.402668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:16.403144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.403144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:16.403144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:16.403144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:16.403144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.403144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:16.403144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:16.403144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:16.403144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:16.403144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.403144 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:16.403390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:16.403668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.403668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:16.403668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.403668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.404325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:16.404636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:16.404972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.404972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:16.404972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.404972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:16.406378 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:16.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:16.406770 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:16.406951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:16.407207 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.407392 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:16.407624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.407974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:16.407974 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.407974 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:16.407974 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:16.407974 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.407974 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:16.407974 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.407974 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:16.408875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.408875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:16.408875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:16.408875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:16.408875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.408875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:16.408875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:16.408875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.408875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:16.408875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:16.408875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:16.408875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.409153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:16.409383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.409383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:16.409999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:16.410324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:16.410624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:16.410624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:16.410624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:21.062124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:21.064487 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:21.064487 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:21.064487 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:21.087380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=243 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:08:21.116046 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.116515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.116910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:21.116910 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.116910 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:21.116910 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:21.116910 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.116910 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:21.116910 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.116910 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:21.117740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.117740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:21.117740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:21.117740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:21.117740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.117740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:21.117740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:21.117740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.117740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:21.117740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:21.117740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:21.117740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.118153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:21.118684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.118684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:21.118883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:21.119397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.119876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:21.119876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.119876 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:21.379387 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:21.379743 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:21.379831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:21.380657 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.380801 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.380913 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.381007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.381123 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.381828 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.381828 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.381828 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.381828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.381975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:21.381975 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.381975 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:21.381975 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:21.381975 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.381975 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:21.381975 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.381975 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:21.383371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.383371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:21.383371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:21.383371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:21.383371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.383371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:21.383371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:21.383371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.383371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:21.383371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:21.383371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:21.383371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.383645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:21.383881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:21.384149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.384672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:21.384672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:21.384672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.384672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:21.384672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.384672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:21.393562 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:21.393859 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:21.394386 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:21.394386 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:08:21.394564 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.394732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.395110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:21.395110 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.395110 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:21.395110 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:21.395110 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.395110 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:21.395110 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.395110 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:21.395618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:21.395931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:21.395931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:21.395931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.395931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:21.395931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:21.395931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.395931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:21.395931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.395931 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:21.396157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.396157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:21.396157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:21.396157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:21.396157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.396157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:21.396157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:21.396157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:08:21.396157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:21.396157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:21.396157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.396157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:21.396385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:21.396611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.396611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:21.396611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.396611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:21.396815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:21.397094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:21.397409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.397409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:21.397409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:21.397409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.397409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:21.397409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.397409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:21.402262 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:21.402262 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:21.402262 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:21.402262 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:21.402262 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.402262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.402530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:21.402909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:21.403203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.403416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:21.403729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:21.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:21.404304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:21.404304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.404304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:21.404304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:21.404304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.404304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:21.404304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.404304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:21.406697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:21.406918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:21.406995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:21.407144 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:21.407409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.407636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.411240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:21.412050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:21.412346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:21.412566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:21.412925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:21.412925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:21.412925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:21.412925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:21.412925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:21.412925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:21.412925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:21.412925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:21.412925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:21.412925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:21.412925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:21.412925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:26.062539 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:26.062976 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:26.062976 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:26.063634 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:26.113932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.113932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.114851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:26.114851 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.114851 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:26.114851 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:26.114851 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.114851 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:26.114851 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.114851 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:26.116071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.052 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.116071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:26.116071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:26.116071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:26.116071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.116071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:26.116071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:26.116071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.116071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:26.116071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:26.116071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:26.116071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.117819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:26.118141 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:26.118501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.119045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:26.119045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:26.119045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.119045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:26.119045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.119045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:26.379799 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:26.380040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:26.380165 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:26.381716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.381908 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.382300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.382518 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.382737 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.382924 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.383161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.383563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.383828 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.384020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.384718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:26.384718 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.384718 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:26.384718 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:26.384718 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.384718 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:26.384718 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.384718 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:26.385815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.385815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:26.385815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:26.385815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:26.385815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.385815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:26.385815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:26.385815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.385815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:26.385815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:26.385815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:26.385815 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.386174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:26.386538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.386538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:26.387321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:26.387749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.388231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:26.388231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.388231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:26.393846 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:26.394052 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:26.394256 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:26.394434 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:08:26.394741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.394941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:26.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:26.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:26.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:26.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.395348 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:26.396059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:26.396629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:26.396629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:26.396629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.396629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:26.396629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:26.396629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.396629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:26.396629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.396629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:26.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:26.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:26.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:26.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:26.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:26.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:08:26.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:26.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:26.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.397448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:26.397779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:26.398234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.398234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:26.398234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.398234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:26.398778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:26.399264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:26.399719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.399719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:26.399719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:26.399719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.399719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:26.399719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.399719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:26.400868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:26.401177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:26.401342 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:26.401501 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:26.403352 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.403602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.404016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:26.404016 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.404016 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:26.404016 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:26.404016 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.404016 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:26.404016 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.404016 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:26.404801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:26.405236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:26.405236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:26.405236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.405236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:26.405236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:26.405236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.405236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:26.405236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.405236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:26.406010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.406010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:26.406010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:26.406010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:26.406010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.406010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:26.406010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:26.406010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:26.406010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:26.406010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.406010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:26.406391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:26.406856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.406856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:26.406856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.406856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.408181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:26.408630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:26.409508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.409508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:26.409508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.409508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:26.409508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:26.409508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:26.409508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:26.409508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:26.410445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.411044 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:26.411431 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.412026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:26.412026 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.412026 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:26.412026 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:26.412026 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.412026 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:26.412026 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.412026 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:26.413469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.413469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:26.413469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:26.413469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:26.413469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.413469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:26.413469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:26.413469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.413469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:26.413469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:26.413469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:26.413469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.413745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:26.414068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.414068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:26.416228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:26.416635 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:26.416973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:26.416973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:26.416973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:31.063435 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:31.063830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:31.063830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:31.064629 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:31.106432 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.106592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.107264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:31.107264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.107264 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:31.107264 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:31.107264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.107264 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:31.107264 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.107264 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:31.107901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.107901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:31.107901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:31.107901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:31.107901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.107901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:31.107901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:31.107901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.107901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:31.107901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:31.107901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:31.107901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.108187 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:31.108548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.108548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:31.108843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:31.109274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.109623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:31.109623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.109623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:31.380236 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:31.383212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:31.383212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:31.383212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.383212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.383212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.383433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.383433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.383433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.383433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.383433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.383551 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.383551 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.383551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.383551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:31.383551 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.383551 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:31.383551 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:31.383551 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.383551 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:31.383551 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.383551 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:31.383944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.383944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:31.383944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:31.383944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:31.383944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.383944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:31.383944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:31.383944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.383944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:31.383944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:31.383944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:31.383944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.384177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:31.384406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.384406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:31.384540 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:31.384854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.385148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:31.385148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.385148 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:31.394640 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:31.394855 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:31.394939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:31.395036 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:08:31.395331 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.395490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.396342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:31.396342 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.396342 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:31.396342 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:31.396342 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.396342 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:31.396342 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.396342 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:31.397005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:31.397324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:31.397324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:31.397324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.397324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:31.397324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:31.397324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.397324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:31.397324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.397324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:31.398123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.398123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:31.398123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:31.398123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:31.398123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.398123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:31.398123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:31.398123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:08:31.398123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:31.398123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:31.398123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.398123 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:31.398442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:31.398671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.398671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:31.398671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.398671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:31.399124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:31.399461 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:31.399854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.399854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:31.399854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:31.399854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.399854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:31.399854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.399854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:31.401240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:31.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:31.401815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:31.401946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:31.403332 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.403631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.404177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:31.404177 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.404177 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:31.404177 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:31.404177 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.404177 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:31.404177 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.404177 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:31.404759 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:31.405027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:31.405027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:31.405027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.405027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:31.405027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:31.405027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.405027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:31.405027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.405027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:31.405571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.405571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:31.405571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:31.405571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:31.405571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.405571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:31.405571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:31.405571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:31.405571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:31.405571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.405571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:31.405784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:31.406033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.406033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:31.406033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.406033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.408229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:31.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:31.410636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.410636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:31.410636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.410636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:31.410636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:31.410636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:31.410636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:31.410636 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:31.411246 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.411246 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:31.411246 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.411515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:31.411515 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.411515 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:31.411515 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:31.411515 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.411515 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:31.411515 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.411515 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:31.412542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.412542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:31.412542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:31.412542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:31.412542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.412542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:31.412542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:31.412542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.412542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:31.412542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:31.412542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:31.412542 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.412933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:31.413328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.413328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:31.414485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:31.414956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:31.415509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:31.415509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:31.415509 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:36.063771 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:36.064296 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:36.064296 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:36.064713 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:36.094471 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.094618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.094986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:36.094986 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.094986 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:36.094986 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:36.094986 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.094986 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:36.094986 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.094986 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:36.095617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.095617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:36.095617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:36.095617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:36.095617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.095617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:36.095617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:36.095617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.095617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:36.095617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:36.095617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:36.095617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.096071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:36.096399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:36.096746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.097129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:36.097129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:36.097129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.097129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:36.097129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.097129 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:36.380682 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:36.381217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:36.381354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:36.383276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.383459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.383635 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.383795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.383947 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.384101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.384468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.384669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.384822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.385119 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.385403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.385930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:36.385930 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.385930 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:36.385930 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:36.385930 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.385930 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:36.385930 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.385930 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:36.386889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.386889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:36.386889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:36.386889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:36.386889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.386889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:36.386889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:36.386889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.386889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:36.386889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:36.386889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:36.386889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.387166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:36.387418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.387418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:36.387987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:36.388359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.388696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:36.388696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.388696 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:36.394984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:36.395488 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:36.395488 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:36.395551 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:08:36.395849 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.396052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.396618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:36.396618 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.396618 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:36.396618 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:36.396618 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.396618 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:36.396618 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.396618 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:36.397403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:36.397723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:36.397723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:36.397723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.397723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:36.397723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:36.397723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.397723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:36.397723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.397723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:36.398273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.398273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:36.398273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:36.398273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:36.398273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.398273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:36.398273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:36.398273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:08:36.398273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:36.398273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:36.398273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.398273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:36.398527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:36.398812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.398812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:36.398812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.398812 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:36.399414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:36.399788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:36.400183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.400183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:36.400183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:36.400183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.400183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:36.400183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.400183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:36.401782 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:36.401997 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:36.402129 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:36.402284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:36.402589 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.402821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.403279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:36.403279 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.403279 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:36.403279 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:36.403279 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.403279 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:36.403279 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.403279 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:36.404015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:36.404331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:36.404331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:36.404331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.404331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:36.404331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:36.404331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.404331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:36.404331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.404331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:36.404941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.404941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:36.404941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:36.404941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:36.404941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.404941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:36.404941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:36.404941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:36.404941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:36.404941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.404941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:36.405166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:36.405520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.405520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:36.405520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.405520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.407562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:36.407906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:36.408212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.408212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:36.408212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.408212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:36.412591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:36.412808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:36.412918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:36.413097 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:36.413386 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.413621 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:36.413930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.417416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:36.417416 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.417416 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:36.417416 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:36.417416 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.417416 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:36.417416 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.417416 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:36.418301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.418301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:36.418301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:36.418301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:36.418301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.418301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:36.418301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:36.418301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.418301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:36.418301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:36.418301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:36.418301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.418612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:36.418854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.418854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:36.419917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:36.420383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:36.420783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:36.420783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:36.420783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:41.064242 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:41.064665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:41.064665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:41.065520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:41.111019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.111235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.111785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:41.111785 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.111785 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:41.111785 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:41.111785 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.111785 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:41.111785 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.111785 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:41.112810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.112810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:41.112810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:41.112810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:41.112810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.112810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:41.112810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:41.112810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.112810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:41.112810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:41.112810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:41.112810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.113082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:41.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:41.113793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:41.114095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.114449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:41.114449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.114449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:41.381402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:41.381713 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:41.381844 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:41.383163 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.383451 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.383514 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.383599 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.383721 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.383825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.383918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.384017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.384140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.384264 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.384370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.384826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:41.384826 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.384826 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:41.384826 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:41.384826 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.384826 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:41.384826 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.384826 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:41.385581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.385581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:41.385581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:41.385581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:41.385581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.385581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:41.385581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:41.385581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.385581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:41.385581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:41.385581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:41.385581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.385940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:41.386297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.386297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:41.386376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:41.386796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.387293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:41.387293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.387293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:41.395430 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:41.395710 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:41.395835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:41.395965 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:08:41.396435 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.396603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.397063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:41.397063 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.397063 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:41.397063 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:41.397063 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.397063 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:41.397063 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.397063 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:41.398023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:41.398562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:41.398562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:41.398562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.398562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:41.398562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:41.398562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.398562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:41.398562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.398562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:41.398909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.398909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:41.398909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:41.398909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:41.398909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.398909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:41.398909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:41.398909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:08:41.398909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:41.398909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:41.398909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.398909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:41.399277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:41.399624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.399624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:41.399624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.399624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:41.399878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:41.400335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:41.400802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.400802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:41.400802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:41.400802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.400802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:41.400802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.400802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:41.402076 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:41.402328 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:41.402431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:41.402571 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:41.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.404026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.404494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:41.404494 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.404494 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:41.404494 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:41.404494 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.404494 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:41.404494 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.404494 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:41.405367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:41.405702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:41.405702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:41.405702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.405702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:41.405702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:41.405702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.405702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:41.405702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.405702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:41.406280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.406280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:41.406280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:41.406280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:41.406280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.406280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:41.406280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:41.406280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:41.406280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:41.406280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.406280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:41.406505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:41.406793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.406793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:41.406793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.406793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.407418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:41.407773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:41.408104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.408104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:41.408104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.408104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:41.412892 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:41.413138 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:41.413275 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:41.413447 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:41.414993 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.415212 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:41.415450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.415928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:41.415928 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.415928 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:41.415928 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:41.415928 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.415928 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:41.415928 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.415928 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:41.417273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.417273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:41.417273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:41.417273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:41.417273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.417273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:41.417273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:41.417273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.417273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:41.417273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:41.417273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:41.417273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.417482 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:41.417750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.417750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:41.417972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:41.418326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:41.418658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:41.418658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:41.418658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:46.067129 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:46.067543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:46.068310 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:46.069864 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:46.109701 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.109701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.110436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:46.110436 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.110436 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:46.110436 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:46.110436 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.110436 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:46.110436 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.110436 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:46.111238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.111238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:46.111238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:46.111238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:46.111238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.111238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:46.111238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:46.111238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.111238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:46.111238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:46.111238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:46.111238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.112182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:46.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:46.113627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.115094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:46.115094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:46.115094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.115094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:46.115094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.115094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:46.382416 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:46.384591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:46.384761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:46.387536 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.387871 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.389240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.389240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.389240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.389240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.389240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.389639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.389639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.389639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.389639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.389639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:46.389639 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.389639 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:46.389639 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:46.389639 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.389639 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:46.389639 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.389639 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:46.390255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.390255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:46.390255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:46.390255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:46.390255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.390255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:46.390255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:46.390255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.390255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:46.390255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:46.390255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:46.390255 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:46.391024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:46.391480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.392018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:46.392018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:46.392018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.392018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:46.392018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.392018 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:46.395756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:46.397409 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:46.397536 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:46.397665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:08:46.398610 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.398610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.398749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:46.398749 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.398749 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:46.398749 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:46.398749 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.398749 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:46.398749 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.398749 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:46.399575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:46.400108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:46.400108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:46.400108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.400108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:46.400108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:46.400108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.400108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:46.400108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.400108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:46.400890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.400890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:46.400890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:46.400890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:46.400890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.400890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:46.400890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:46.400890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:08:46.400890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:46.400890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:46.400890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.400890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:46.401347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:46.401742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.401742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:46.401742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.401742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:46.402569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:46.403365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:46.403939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.403939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:46.403939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:46.403939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.403939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:46.403939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.403939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:46.403939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:46.403939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:46.404280 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:46.405168 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:46.405168 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.405168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.405448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:46.405448 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.405448 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:46.405448 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:46.405448 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.405448 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:46.405448 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.405448 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:46.406388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:46.407068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:46.407068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:46.407068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.407068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:46.407068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:46.407068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.407068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:46.407068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.407068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:46.407549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.407549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:46.407549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:46.407549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:46.407549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.407549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:46.407549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:46.407549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:46.407549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:46.407549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.407549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:46.407862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:46.408233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.408233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:46.408233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.408233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.408607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:46.409962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:46.410860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.410860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:46.410860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.410860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:46.413244 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:46.413410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:46.413528 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:46.413701 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:46.414863 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.414863 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:46.414863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.415107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:46.415107 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.415107 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:46.415107 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:46.415107 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.415107 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:46.415107 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.415107 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:46.416173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.416173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:46.416173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:46.416173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:46.416173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.416173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:46.416173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:46.416173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.416173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:46.416173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:46.416173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:46.416173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:46.416975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.416975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:46.417693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:46.418553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:46.419416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:46.419416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:46.419416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:51.067544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:51.068037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:51.068143 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:51.068763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:51.110034 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.110266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.111159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:51.111159 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.111159 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:51.111159 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:51.111159 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.111159 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:51.111159 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.111159 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:51.111487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.111487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:51.111487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:51.111487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:51.111487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.111487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:51.111487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:51.111487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.111487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:51.111487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:51.111487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:51.111487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.111897 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:51.112333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:51.113380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.114126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:51.114126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:51.114126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.114126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:51.114126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.114126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:51.386995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:51.387238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:51.387343 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:51.388374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.388554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.388664 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.388807 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.388928 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.389043 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.389175 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.389318 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.389423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.389519 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.389649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.390119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:51.390119 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.390119 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:51.390119 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:51.390119 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.390119 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:51.390119 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.390119 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:51.390963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.390963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:51.390963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:51.390963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:51.390963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.390963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:51.390963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:51.390963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.390963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:51.390963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:51.390963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:51.390963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.391229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:51.391459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.391459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:51.391661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:51.391956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.392274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:51.392274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.392274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:51.397487 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:51.397765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:51.397855 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:51.397959 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:08:51.399513 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.399713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.400352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:51.400352 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.400352 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:51.400352 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:51.400352 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.400352 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:51.400352 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.400352 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:51.401127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:51.401454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:51.401454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:51.401454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.401454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:51.401454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:51.401454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.401454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:51.401454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.401454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:51.401822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.401822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:51.401822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:51.401822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:51.401822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.401822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:51.401822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:51.401822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:08:51.401822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:51.401822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:51.401822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.401822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:51.402105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:51.402417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.402417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:51.402417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.402417 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:51.402677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:51.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:51.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:51.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:51.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:51.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:51.403330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:51.403492 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:51.403492 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:51.403593 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:51.403860 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.404102 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.404598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:51.404598 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.404598 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:51.404598 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:51.404598 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.404598 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:51.404598 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.404598 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:51.405214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:51.405589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:51.405589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:51.405589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.405589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:51.405589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:51.405589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.405589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:51.405589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.405589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:51.406139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.406139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:51.406139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:51.406139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:51.406139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.406139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:51.406139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:51.406139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:51.406139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:51.406139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.406139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:51.406420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:51.406685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.406685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:51.406685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.406685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.406962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:51.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:51.407802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.407802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:51.407802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.407802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:51.413544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:51.413832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:51.413923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:51.414051 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:51.414934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.415183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:51.415519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.416011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:51.416011 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.416011 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:51.416011 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:51.416011 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.416011 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:51.416011 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.416011 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:51.416769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.416769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:51.416769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:51.416769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:51.416769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.416769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:51.416769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:51.416769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.416769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:51.416769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:51.416769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:51.416769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.416997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:51.417260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.417260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:51.417706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:51.417996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:51.418568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:51.418568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:51.418568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:56.067983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:56.068508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:56.068508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:56.069059 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:56.104228 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.104228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.104602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:56.104602 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.104602 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:56.104602 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:56.104602 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.104602 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:56.104602 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.104602 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:56.105061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.105061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:56.105061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:56.105061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:56.105061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.105061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:56.105061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:56.105061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.105061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:56.105061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:56.105061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:56.105061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.105415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:56.105688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.105688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:56.105748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:56.106109 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.106535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:56.106535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.106535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:56.387413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:56.387763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:56.387868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:56.388943 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.389141 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.389303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.389453 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.389602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.389741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.389927 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.390032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.390166 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.390299 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.390407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.390860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:56.390860 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.390860 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:56.390860 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:56.390860 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.390860 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:56.390860 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.390860 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:56.391592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.391592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:56.391592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:56.391592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:56.391592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.391592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:56.391592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:56.391592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.391592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:56.391592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:56.391592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:56.391592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.391833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:56.392036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.392036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:56.392323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:56.392630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.392945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:56.392945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.392945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:56.397871 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:56.398132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:56.398250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:56.398349 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:08:56.398742 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.398913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.399331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:56.399331 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.399331 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:56.399331 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:56.399331 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.399331 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:56.399331 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.399331 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:56.399901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:56.400212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:56.400212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:56.400212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.400212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:56.400212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:56.400212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.400212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:56.400212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.400212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:56.400495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.400495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:56.400495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:56.400495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:56.400495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.400495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:56.400495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:56.400495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:08:56.400495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:56.400495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:56.400495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.400495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:56.400741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:56.400956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.400956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:56.400956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.400956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:56.401225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:56.401513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:56.401825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.401825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:56.401825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:56.401825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.401825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:56.401825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.401825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:56.403498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:56.403736 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:56.403821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:56.403918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:56.404248 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.404428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:56.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:56.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:56.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:56.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:08:56.405336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:08:56.405748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:56.405748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:08:56.405748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.405748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:08:56.405748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:08:56.405748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.405748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:08:56.405748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.405748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:08:56.405991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.405991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:56.405991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:56.405991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:56.405991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.405991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:56.405991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:56.405991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:56.405991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:56.405991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.405991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:56.406279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:56.406519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.406519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:56.406519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.406519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.406718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:56.407034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:56.407386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.407386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:56.407386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.407386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:08:56.413922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:08:56.414160 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:08:56.414276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:08:56.414410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:08:56.414756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.414936 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:08:56.415175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.415597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:08:56.415597 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.415597 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:08:56.415597 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:08:56.415597 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.415597 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:08:56.415597 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.415597 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:08:56.416237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.416237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:08:56.416237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:56.416237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:08:56.416237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.416237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:56.416237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:56.416237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.416237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:56.416237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:56.416237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:08:56.416237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.416493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:08:56.416714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.416714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:08:56.416883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:08:56.417180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:08:56.417511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:08:56.417511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:08:56.417511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:01.068444 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:01.068981 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:01.068981 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:01.069665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:01.100478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.100741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.101373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:01.101373 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.101373 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:01.101373 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:01.101373 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.101373 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:01.101373 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.101373 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:01.102261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.102261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:01.102261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:01.102261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:01.102261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.102261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:01.102261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:01.102261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.102261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:01.102261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:01.102261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:01.102261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.102853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:01.103529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:01.103813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.104243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:01.104243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:01.104243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.104243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:01.104243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.104243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:01.387979 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:01.388328 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:01.388419 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:01.389547 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.389742 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.389884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.389990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.390129 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.390276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.390388 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.390497 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.390639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.390760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.390889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.393294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:01.393294 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.393294 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:01.393294 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:01.393294 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.393294 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:01.393294 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.393294 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:01.394355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.394355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:01.394355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:01.394355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:01.394355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.394355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:01.394355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:01.394355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.394355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:01.394355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:01.394355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:01.394355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.394584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:01.394828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.394828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:01.395145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:01.395454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.395769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:01.395769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.395769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:01.398341 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:01.399431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:01.399563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:01.399684 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:09:01.400673 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.400673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.400958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:01.400958 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.400958 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:01.400958 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:01.400958 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.400958 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:01.400958 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.400958 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:01.401817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:01.402201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:01.402201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:01.402201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.402201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:01.402201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:01.402201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.402201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:01.402201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.402201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:01.402778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.402778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:01.402778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:01.402778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:01.402778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.402778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:01.402778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:01.402778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:09:01.402778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:01.402778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:01.402778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.402778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:01.403274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:01.403717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.403717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:01.403717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.403717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:01.404055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:01.404565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:01.405076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.405076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:01.405076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:01.405076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.405076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:01.405076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.405076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:01.405076 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:01.405076 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:01.405076 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:01.405374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:01.405374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.406119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.406119 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:01.406119 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.406119 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:01.406119 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:01.406119 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.406119 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:01.406119 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.406119 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:01.407128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:01.407943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:01.407943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:01.407943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.407943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:01.407943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:01.407943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.407943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:01.407943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.407943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:01.408507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.408507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:01.408507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:01.408507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:01.408507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.408507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:01.408507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:01.408507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:01.408507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:01.408507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.408507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:01.409072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:01.410058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:01.410961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:01.412015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:01.412015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:01.412015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:01.412015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:01.412015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.412015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:01.412015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:01.412015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.412015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:01.412015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.412015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:01.414379 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:01.415361 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:01.415492 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:01.415643 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:01.416785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.417203 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:01.417584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.418270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:01.418270 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.418270 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:01.418270 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:01.418270 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.418270 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:01.418270 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.418270 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:01.419120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.419120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:01.419120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:01.419120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:01.419120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.419120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:01.419120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:01.419120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.419120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:01.419120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:01.419120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:01.419120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:01.419943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.419943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:01.420289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:01.421179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:01.422276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:01.422276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:01.422276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:06.068845 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:06.069588 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:06.069588 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:06.070168 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:06.111030 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.111228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.111720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:06.111720 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.111720 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:06.111720 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:06.111720 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.111720 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:06.111720 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.111720 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:06.112406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.112406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:06.112406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:06.112406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:06.112406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.112406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:06.112406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:06.112406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.112406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:06.112406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:06.112406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:06.112406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.112790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:06.113153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.113153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:06.113333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:06.114551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.115409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:06.115409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.115409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:06.388496 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:06.388698 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:06.388817 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:06.390047 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.390271 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.390394 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.390501 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.390613 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.390709 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.390808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.390906 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.391001 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.391107 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.391234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.391703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:06.391703 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.391703 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:06.391703 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:06.391703 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.391703 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:06.391703 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.391703 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:06.392502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.392502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:06.392502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:06.392502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:06.392502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.392502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:06.392502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:06.392502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.392502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:06.392502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:06.392502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:06.392502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.392777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:06.392990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.392990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:06.393284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:06.393560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.393878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:06.393878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.393878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:06.399421 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:06.399648 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:06.399765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:06.399885 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:09:06.400265 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.400455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.400910 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:06.400910 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.400910 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:06.400910 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:06.400910 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.400910 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:06.400910 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.400910 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:06.401572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:06.402352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:06.402825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:06.403110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:06.403110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:06.403110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:06.403110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:06.403110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:06.403110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.403110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:06.403110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:06.403110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.403110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:06.403110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.403110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:06.403342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:06.403595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:06.403877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.403877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:06.403877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:06.403877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.403877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:06.403877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.403877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:06.405611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:06.405805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:06.405935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:06.406321 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:06.406663 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.406861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.407456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:06.407456 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.407456 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:06.407456 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:06.407456 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.407456 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:06.407456 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.407456 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:06.408125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:06.408673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:06.408673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:06.408673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.408673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:06.408673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:06.408673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.408673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:06.408673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.408673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:06.409074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.409074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:06.409074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:06.409074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:06.409074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.409074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:06.409074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:06.409074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:06.409074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:06.409074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.409074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:06.409515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:06.409877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.409877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:06.409877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.409877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.410149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:06.410464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:06.410733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.410733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:06.410733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.410733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:06.415380 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:06.415653 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:06.415749 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:06.415915 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:06.416169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.416413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:06.416646 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.417135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:06.417135 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.417135 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:06.417135 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:06.417135 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.417135 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:06.417135 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.417135 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:06.417780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.417780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:06.417780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:06.417780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:06.417780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.417780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:06.417780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:06.417780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.417780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:06.417780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:06.417780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:06.417780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.418005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:06.418239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.418239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:06.418455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:06.418710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:06.418982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:06.418982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:06.418982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:11.071269 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:11.071269 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:11.071269 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:11.071269 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:11.112942 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.113225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.113791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:11.113791 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.113791 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:11.113791 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:11.113791 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.113791 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:11.113791 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.113791 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:11.114630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.114630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:11.114630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:11.114630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:11.114630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.114630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:11.114630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:11.114630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.114630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:11.114630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:11.114630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:11.114630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.115528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:11.116327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:11.117223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.120346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:11.120346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:11.120346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.120346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:11.120346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.120346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:11.388871 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:11.389240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:11.389364 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:11.390451 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.390631 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.390770 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.390926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.391057 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.391202 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.391316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.391414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.391517 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.391595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.391700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.392290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:11.392290 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.392290 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:11.392290 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:11.392290 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.392290 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:11.392290 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.392290 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:11.393208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.393208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:11.393208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:11.393208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:11.393208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.393208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:11.393208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:11.393208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.393208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:11.393208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:11.393208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:11.393208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.393973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:11.394819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:11.395802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.396922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:11.396922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:11.396922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.396922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:11.396922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.396922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:11.399830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:11.400283 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:11.400412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:11.400543 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:09:11.400920 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.401103 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.401612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:11.401612 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.401612 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:11.401612 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:11.401612 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.401612 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:11.401612 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.401612 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:11.402344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:11.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:11.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:11.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:11.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:11.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:11.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.402728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:11.403236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.403236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:11.403236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:11.403236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:11.403236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.403236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:11.403236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:11.403236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:09:11.403236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:11.403236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:11.403236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.403236 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:11.403977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:11.404329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:11.404666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:11.405110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:11.405110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:11.405110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:11.405110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:11.405110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:11.405110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.405110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:11.405110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:11.405110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.405110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:11.405110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.405110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:11.406000 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:11.406224 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:11.406318 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:11.406395 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:11.406743 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.406945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.407406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:11.407406 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.407406 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:11.407406 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:11.407406 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.407406 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:11.407406 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.407406 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:11.407957 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:11.408898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:11.409579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:11.410214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:11.410214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:11.410214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:11.410214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:11.410214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.410214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:11.410214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:11.410214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.410214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:11.410214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.410214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:11.410214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:11.410493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.410919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:11.419418 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:11.429111 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:11.429317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:11.429435 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:11.435556 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.435781 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:11.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.436579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:11.436579 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.436579 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:11.436579 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:11.436579 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.436579 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:11.436579 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.436579 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:11.437337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.437337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:11.437337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:11.437337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:11.437337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.437337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:11.437337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:11.437337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.437337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:11.437337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:11.437337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:11.437337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.437575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:11.437764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.437764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:11.438065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:11.438368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:11.438655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:11.438655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:11.438655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:16.298413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:16.298413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:16.298413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:16.298413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:16.298413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.298413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.101 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:16.318055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:16.324133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.329120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:16.330443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:16.333620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:16.333620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:16.333620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.333620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:16.333620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:16.333620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.333620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:16.333620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.333620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:16.389264 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:16.389829 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:16.389829 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:16.390590 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.390770 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.391860 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.391860 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.391860 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.391860 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.391860 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.394785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.394785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.394785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e56abc3-0f43-47f4-9602-66084910993c, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.394785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.394785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:16.394785 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.394785 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:16.394785 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:16.394785 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.394785 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:16.394785 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.394785 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:16.396912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.396912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:16.396912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:16.396912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:16.396912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.396912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:16.396912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:16.396912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.396912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:16.396912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:16.396912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:16.396912 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.397729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:16.398140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:16.398711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.399284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:16.399284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:16.399284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.399284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:16.399284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.399284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:16.400239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:16.400460 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:16.400547 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:16.400638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:09:16.400937 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.401133 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.401502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:16.401502 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.401502 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:16.401502 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:16.401502 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.401502 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:16.401502 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.401502 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:16.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:16.403467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:16.403467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:16.403467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.403467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:16.403467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:16.403467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.403467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:16.403467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.403467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:16.404622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.404622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:16.404622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:16.404622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:16.404622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.404622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:16.404622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:16.404622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:09:16.404622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:16.404622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:16.404622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.404622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:16.405470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:16.405851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:16.406295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:16.407154 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:16.409367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:16.409443 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:16.409551 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:16.409825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.410005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:16.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:16.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:16.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:16.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:16.410848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:16.411318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:16.411318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:16.411318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.411318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:16.411318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:16.411318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.411318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:16.411318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.411318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:16.411571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.411571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:16.411571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:16.411571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:16.411571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.411571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:16.411571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:16.411571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:16.411571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:16.411571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.411571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:16.412952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:16.413214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:16.413459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:16.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:16.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:16.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:16.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:16.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:16.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:16.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:16.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:16.429112 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:16.429394 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:16.429475 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:16.429616 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:16.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.430153 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:16.430410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.430809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:16.430809 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.430809 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:16.430809 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:16.430809 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.430809 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:16.430809 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.430809 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:16.431519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.431519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:16.431519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:16.431519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:16.431519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.431519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:16.431519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:16.431519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.431519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:16.431519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:16.431519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:16.431519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.432590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:16.432875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:16.433230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:16.433587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:16.433587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:16.433587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:16.433587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:16.433587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:16.433587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:21.071955 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:21.076225 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:21.076225 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:21.076225 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:21.092525 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=241 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:09:21.112631 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.112631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.113152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:21.113152 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.113152 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:21.113152 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:21.113152 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.113152 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:21.113152 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.113152 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:21.114514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.114514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:21.114514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:21.114514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:21.114514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.114514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:21.114514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:21.114514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.114514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:21.114514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:21.114514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:21.114514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.116856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:21.117181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:21.118367 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.118814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:21.118814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:21.118814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.118814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:21.118814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.118814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:21.389673 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:21.389894 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:21.390054 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:21.392130 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.392130 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.392958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.392958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.392958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.392958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.392958 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.395418 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.395418 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.395418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.395418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:21.395418 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.395418 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:21.395418 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:21.395418 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.395418 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:21.395418 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.395418 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:21.395418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:21.398978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:21.399463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:21.399463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:21.399463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:21.399463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:21.399463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:21.399463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.399463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:21.399463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:21.399463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.399463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:21.399463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.399463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:21.399930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:21.400528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.401253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:21.401253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.401253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:21.401253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:21.401253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:21.401253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:21.401253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:09:21.401716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.401887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.402401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:21.402401 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.402401 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:21.402401 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:21.402401 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.402401 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:21.402401 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.402401 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:21.403375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:21.404305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:21.404743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:21.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:21.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:21.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:21.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:21.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:21.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:21.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:21.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:21.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.405475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:21.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:21.405994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:21.406344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.406344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:21.406344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:21.406344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.406344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:21.406344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.406344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:21.409433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:21.409640 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:21.409755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:21.409887 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:21.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.410617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.411073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:21.411073 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.411073 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:21.411073 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:21.411073 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.411073 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:21.411073 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.411073 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:21.412726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:21.413003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:21.413380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:21.413607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:21.413607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:21.413607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:21.413607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:21.413607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.413607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:21.413607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:21.413607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.413607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:21.413607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.413607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.413969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:21.414300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:21.414616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.414616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:21.414616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.414616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:21.429479 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:21.429707 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:21.429790 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:21.429927 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:21.430390 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.430597 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:21.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.431314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:21.431314 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.431314 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:21.431314 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:21.431314 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.431314 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:21.431314 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.431314 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:21.431938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.431938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:21.431938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:21.431938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:21.431938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.431938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:21.431938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:21.431938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.431938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:21.431938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:21.431938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:21.431938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.432213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:21.432436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.432436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:21.432611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:21.432905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:21.433231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:21.433231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:21.433231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:26.075215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:26.075215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:26.075215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:26.075215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:26.114028 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.114254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.114790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:26.114790 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.114790 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:26.114790 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:26.114790 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.114790 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:26.114790 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.114790 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:26.115550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.115550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:26.115550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:26.115550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:26.115550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.115550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:26.115550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:26.115550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.115550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:26.115550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:26.115550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:26.115550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.116893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:26.117284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:26.117707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.118286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:26.118286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:26.118286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.118286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:26.118286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.118286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:26.390530 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:26.390849 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:26.390982 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:26.392884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.393055 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.393227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.393374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.393475 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.393582 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.393708 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.393808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.393903 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.394000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.394489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:26.394489 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.394489 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:26.394489 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:26.394489 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.394489 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:26.394489 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.394489 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:26.395472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.395472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:26.395472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:26.395472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:26.395472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.395472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:26.395472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:26.395472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.395472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:26.395472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:26.395472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:26.395472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.395721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:26.395946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.395946 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:26.396728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:26.397038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.397394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:26.397394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.397394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:26.400869 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:26.401073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:26.401235 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:26.401358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:09:26.403483 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.403685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.404307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:26.404307 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.404307 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:26.404307 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:26.404307 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.404307 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:26.404307 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.404307 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:26.405030 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:26.405364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:26.405364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:26.405364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.405364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:26.405364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:26.405364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.405364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:26.405364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.405364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:26.405802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.405802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:26.405802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:26.405802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:26.405802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.405802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:26.405802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:26.405802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:09:26.405802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:26.405802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:26.405802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.405802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:26.406609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:26.407011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:26.407427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:26.407939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:26.407939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:26.407939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:26.407939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:26.407939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:26.407939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.407939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:26.407939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:26.407939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.407939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:26.407939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.407939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:26.409761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:26.410044 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:26.410167 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:26.410317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:26.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.410778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.411220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:26.411220 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.411220 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:26.411220 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:26.411220 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.411220 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:26.411220 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.411220 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:26.411952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:26.412421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:26.412421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:26.412421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.412421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:26.412421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:26.412421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.412421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:26.412421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.412421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:26.412927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.412927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:26.412927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:26.412927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:26.412927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.412927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:26.412927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:26.412927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:26.412927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:26.412927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.412927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:26.413282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:26.413655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.413655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:26.413655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.413655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.414039 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:26.414416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:26.414772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.414772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:26.414772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.414772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:26.430498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:26.430498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:26.430498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:26.430498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:26.432777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.432777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:26.432777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.433421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:26.433421 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.433421 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:26.433421 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:26.433421 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.433421 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:26.433421 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.433421 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:26.434036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.434036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:26.434036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:26.434036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:26.434036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.434036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:26.434036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:26.434036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.434036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:26.434036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:26.434036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:26.434036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.434883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:26.435276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:26.435779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:26.436376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:26.436376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:26.436376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:26.436376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:26.436376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:26.436376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:31.073432 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:31.073993 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:31.073993 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:31.074375 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:31.108641 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.108641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.108641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:31.108641 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.108641 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:31.108641 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:31.108641 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.108641 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:31.108641 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.108641 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:31.109285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.109285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:31.109285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:31.109285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:31.109285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.109285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:31.109285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:31.109285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.109285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:31.109285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:31.109285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:31.109285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.110380 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:31.110841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:31.111442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.112098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:31.112098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:31.112098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.112098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:31.112098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.112098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:31.392649 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:31.396607 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:31.396719 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:31.398217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.398217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.398217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.398217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.398217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.398397 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.398510 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.398697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.398918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.399167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.400159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:31.400159 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.400159 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:31.400159 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:31.400159 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.400159 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:31.400159 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.400159 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:31.400774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.400774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:31.400774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:31.400774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:31.400774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.400774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:31.400774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:31.400774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.400774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:31.400774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:31.400774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:31.400774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.401676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:31.402077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:31.402484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.402847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:31.402847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:31.402847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.402847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:31.402847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.402847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:31.402847 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:31.402847 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:31.402847 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:31.402847 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:09:31.403638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.403638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.403757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:31.403757 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.403757 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:31.403757 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:31.403757 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.403757 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:31.403757 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.403757 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:31.404489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:31.404777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:31.404777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:31.404777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.404777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:31.404777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:31.404777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.404777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:31.404777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.404777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:31.405312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.405312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:31.405312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:31.405312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:31.405312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.405312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:31.405312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:31.405312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:09:31.405312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:31.405312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:31.405312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.405312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:31.405618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:31.405896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.405896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:31.405896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.405896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:31.406336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:31.406615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:31.406919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.406919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:31.406919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:31.406919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.406919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:31.406919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.406919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:31.410204 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:31.410552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:31.410683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:31.410861 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:31.411677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.411677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.411859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:31.411859 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.411859 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:31.411859 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:31.411859 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.411859 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:31.411859 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.411859 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:31.412675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:31.413074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:31.413074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:31.413074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.413074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:31.413074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:31.413074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.413074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:31.413074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.413074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:31.413490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.413490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:31.413490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:31.413490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:31.413490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.413490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:31.413490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:31.413490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:31.413490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:31.413490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.413490 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:31.413786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:31.414146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.414146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:31.414146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.414146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.414653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:31.415162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:31.415705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.415705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:31.415705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.415705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:31.430358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:31.430760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:31.430947 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:31.431167 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:31.432650 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.432650 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:31.432650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.432834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:31.432834 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.432834 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:31.432834 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:31.432834 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.432834 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:31.432834 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.432834 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:31.433598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.433598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:31.433598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:31.433598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:31.433598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.433598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:31.433598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:31.433598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.433598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:31.433598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:31.433598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:31.433598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:31.434285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.434285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:31.434351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:31.434653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:31.435680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:31.435680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:31.435680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:36.073816 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:36.074276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:36.074276 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:36.074812 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:36.108349 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.108505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.108901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:36.108901 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.108901 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:36.108901 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:36.108901 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.108901 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:36.108901 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.108901 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:36.109521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.109521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:36.109521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:36.109521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:36.109521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.109521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:36.109521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:36.109521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.109521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:36.109521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:36.109521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:36.109521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.110167 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:36.110394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:36.110654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.110954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:36.110954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:36.110954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.110954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:36.110954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.110954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:36.396740 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:36.396956 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:36.397173 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:36.397906 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.398294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.398294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.398294 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.398427 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.398548 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.398548 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.399004 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.399004 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.399004 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.399004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.399881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:36.399881 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.399881 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:36.399881 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:36.399881 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.399881 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:36.399881 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.399881 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:36.400184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.400184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:36.400184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:36.400184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:36.400184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.400184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:36.400184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:36.400184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.400184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:36.400184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:36.400184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:36.400184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.401045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:36.401480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:36.401913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.402587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:36.402587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:36.402587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.402587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:36.402587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.402587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:36.402587 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:36.402783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:36.402783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:36.402783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:09:36.403169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.403254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:36.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:36.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:36.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:36.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.403616 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:36.404110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:36.404837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:36.405316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:36.405900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:36.405900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:36.405900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:36.405900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:36.405900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:36.405900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.405900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:36.405900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:36.405900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.405900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:36.405900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.405900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:36.406407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:36.406934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:36.407415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.407415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:36.407415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:36.407415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.407415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:36.407415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.407415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:36.412504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:36.412504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:36.412504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:36.412504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:36.412504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.412504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.413383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:36.413901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:36.414362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.414731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:36.415115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:36.415563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:36.416585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:36.416585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.416585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:36.416585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:36.416585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.416585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:36.416585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.416585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:36.430880 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:36.431172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:36.431436 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:36.431506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:36.433555 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.433760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:36.433988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.434519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:36.434519 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.434519 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:36.434519 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:36.434519 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.434519 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:36.434519 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.434519 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:36.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:36.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:36.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:36.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:36.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:36.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:36.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:36.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:36.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.435824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:36.436047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.436047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:36.436645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:36.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:36.437291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:36.437291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:36.437291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:41.074208 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:41.074870 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:41.074870 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:41.075375 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:41.107589 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.107589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.107884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:41.107884 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.107884 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:41.107884 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:41.107884 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.107884 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:41.107884 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.107884 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:41.108456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.108456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:41.108456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:41.108456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:41.108456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.108456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:41.108456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:41.108456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.108456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:41.108456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:41.108456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:41.108456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.108836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:41.109249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:41.109737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:41.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:41.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:41.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.110344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:41.397101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:41.397591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:41.397708 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:41.400279 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.400279 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.400279 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.400279 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.400279 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.400550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.400550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.400550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.400550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.400550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.400710 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.400710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.400710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:41.400710 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.400710 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:41.400710 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:41.400710 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.400710 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:41.400710 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.400710 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:41.401016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.401016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:41.401016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:41.401016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:41.401016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.401016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:41.401016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:41.401016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.401016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:41.401016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:41.401016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:41.401016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.401269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:41.401538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.401538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:41.401702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:41.401977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.402306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:41.402306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.402306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:41.406410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:41.406641 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:41.406720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:41.406813 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:09:41.407464 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.407464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.407583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:41.407583 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.407583 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:41.407583 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:41.407583 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.407583 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:41.407583 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.407583 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:41.408075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:41.408705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:41.409225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:41.409714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:41.409714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:41.409714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:41.409714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:41.409714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:41.409714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.409714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:41.409714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:41.409714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.409714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:41.409714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.409714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:41.410120 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:41.410695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:41.411413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.411413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:41.411413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:41.411413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.411413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:41.411413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.411413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:41.411413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:41.411413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:41.411413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:41.411595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:41.412301 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.412301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.412429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:41.412429 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.412429 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:41.412429 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:41.412429 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.412429 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:41.412429 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.412429 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:41.412848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:41.413403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:41.413403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:41.413403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.413403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:41.413403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:41.413403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.413403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:41.413403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.413403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:41.413711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.413711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:41.413711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:41.413711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:41.413711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.413711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:41.413711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:41.413711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:41.413711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:41.413711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.413711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:41.413992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:41.414280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:41.414513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:41.414811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:41.414811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:41.414811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:41.414811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:41.414811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.414811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:41.414811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:41.414811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.414811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:41.414811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.414811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:41.431283 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:41.431584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:41.431660 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:41.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:41.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:41.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.432990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:41.432990 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.432990 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:41.432990 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:41.432990 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.432990 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:41.432990 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.432990 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:41.433438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.433438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:41.433438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:41.433438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:41.433438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.433438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:41.433438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:41.433438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.433438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:41.433438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:41.433438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:41.433438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.434020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:41.434353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:41.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:41.434919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:41.434919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:41.434919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:41.434919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:41.434919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:41.434919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:46.074603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:46.074969 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:46.075035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:46.075700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:46.108841 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.109032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.109651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:46.109651 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.109651 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:46.109651 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:46.109651 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.109651 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:46.109651 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.109651 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:46.110354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.110354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:46.110354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:46.110354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:46.110354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.110354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:46.110354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:46.110354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.110354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:46.110354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:46.110354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:46.110354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.110598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:46.110809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.110809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:46.111046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:46.111355 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.111663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:46.111663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.111663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:46.397589 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:46.397944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:46.398037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:46.399011 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.399207 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.399341 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.399448 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.399544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.399645 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.399742 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.399857 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.399952 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.400051 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.400161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.400280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.400708 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:46.400708 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.400708 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:46.400708 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:46.400708 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.400708 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:46.400708 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.400708 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:46.401424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.401424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:46.401424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:46.401424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:46.401424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.401424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:46.401424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:46.401424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.401424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:46.401424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:46.401424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:46.401424 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.401793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:46.402207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:46.402669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.403241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:46.403241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:46.403241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.403241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:46.403241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.403241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:46.406716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:46.406914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:46.406983 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:46.407079 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:09:46.407363 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.407502 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:46.408377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:46.408829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:46.408829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:46.408829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.408829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:46.408829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:46.408829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.408829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:46.408829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.408829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:46.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:46.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:46.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:46.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:46.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:46.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:09:46.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:46.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:46.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:46.409505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:46.409916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:46.410365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:46.411012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:46.411012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:46.411012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:46.411012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:46.411012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:46.411012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.411012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:46.411012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:46.411012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.411012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:46.411012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.411012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:46.411377 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:46.411442 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:46.411527 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:46.411632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:46.411894 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.412046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.412471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:46.412471 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.412471 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:46.412471 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:46.412471 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.412471 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:46.412471 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.412471 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:46.412973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:46.413412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:46.413412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:46.413412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.413412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:46.413412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:46.413412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.413412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:46.413412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.413412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:46.413668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.413668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:46.413668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:46.413668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:46.413668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.413668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:46.413668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:46.413668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:46.413668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:46.413668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.413668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:46.413970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:46.414366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:46.414729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:46.415235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:46.415235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:46.415235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:46.415235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:46.415235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.415235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:46.415235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:46.415235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.415235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:46.415235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.415235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:46.431763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:46.432140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:46.432268 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:46.432441 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:46.432744 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.432928 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:46.433175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.433619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:46.433619 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.433619 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:46.433619 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:46.433619 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.433619 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:46.433619 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.433619 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:46.434432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.434432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:46.434432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:46.434432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:46.434432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.434432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:46.434432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:46.434432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.434432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:46.434432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:46.434432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:46.434432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.434694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:46.434905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.434905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:46.435060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:46.435350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:46.435645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:46.435645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:46.435645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:51.074995 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:51.075396 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:51.075396 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:51.075929 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:51.111017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.111234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.111959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:51.111959 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.111959 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:51.111959 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:51.111959 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.111959 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:51.111959 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.111959 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:51.112598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.112598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:51.112598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:51.112598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:51.112598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.112598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:51.112598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:51.112598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.112598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:51.112598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:51.112598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:51.112598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.112836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:51.113079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.113079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:51.113331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:51.113927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.114718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:51.114718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.114718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:51.397956 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:51.398499 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:51.398591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:51.399514 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.399675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.399800 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.399932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d78b2d36-57fe-448e-8a41-91df179ffd25, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.400069 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.400298 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.400363 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.400476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.400602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.400722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.400844 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-dc4f8927-404b-4d88-98ae-91a006596016, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.400978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.401504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:51.401504 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.401504 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:51.401504 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:51.401504 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.401504 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:51.401504 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.401504 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:51.402083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.402083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:51.402083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:51.402083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:51.402083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.402083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:51.402083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:51.402083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.402083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:51.402083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:51.402083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:51.402083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.402415 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:51.402647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.402647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:51.402932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:51.403280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.403632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:51.403632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.403632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:51.407307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:51.407519 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:51.407603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:51.407703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:09:51.407969 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.408145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.408653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:51.408653 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.408653 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:51.408653 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:51.408653 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.408653 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:51.408653 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.408653 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:51.409226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:51.409549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:51.409549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:51.409549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.409549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:51.409549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:51.409549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.409549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:51.409549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.409549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:51.409861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.409861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:51.409861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:51.409861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:51.409861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.409861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:51.409861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:51.409861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:09:51.409861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:51.409861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:51.409861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.409861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:51.410151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:51.410427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.410427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:51.410427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.410427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:51.410648 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:51.410955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:51.411313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.411313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:51.411313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:51.411313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.411313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:51.411313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.411313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:51.411563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:51.411694 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:51.411774 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:51.411928 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:51.412160 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.412401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.412811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:51.412811 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.412811 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:51.412811 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:51.412811 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.412811 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:51.412811 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.412811 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:51.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:51.413633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:51.413633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:51.413633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.413633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:51.413633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:51.413633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.413633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:51.413633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.413633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:51.413938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.413938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:51.413938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:51.413938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:51.413938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.413938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:51.413938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:51.413938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:51.413938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:51.413938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.413938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:51.414183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:51.414471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.414471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:51.414471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.414471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.414720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:51.415152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:51.415471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.415471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:51.415471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.415471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:51.432166 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:51.432602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:51.432655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:51.432788 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:51.433153 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.433388 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:51.433650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.434076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:51.434076 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.434076 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:51.434076 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:51.434076 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.434076 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:51.434076 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.434076 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:51.434771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.434771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:51.434771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:51.434771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:51.434771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.434771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:51.434771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:51.434771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.434771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:51.434771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:51.434771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:51.434771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.435010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:51.435608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.435608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:51.435675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:51.436003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:51.436411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:51.436411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:51.436411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:56.075411 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:56.075938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:56.075938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:56.076632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:56.128997 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.129180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.129694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:56.129694 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.129694 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:56.129694 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:56.129694 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.129694 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:56.129694 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.129694 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:56.130800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.130800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:56.130800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:56.130800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:56.130800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.130800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:56.130800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:56.130800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.130800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:56.130800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:56.130800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:56.130800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.131694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:56.132765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:56.133658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:56.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:56.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:56.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.134531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:56.398922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:56.399336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:56.399573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:56.400438 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.401938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.401938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.401938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.401938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.401938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.402124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.402124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.402124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.402124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.402124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:56.402124 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.402124 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:56.402124 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:56.402124 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.402124 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:56.402124 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.402124 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:56.402709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.402709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:56.402709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:56.402709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:56.402709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.402709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:56.402709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:56.402709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.402709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:56.402709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:56.402709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:56.402709 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.402980 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:56.403240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.403240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:56.403397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:56.403676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.404003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:56.404003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.404003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:56.407640 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:56.407875 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:56.407971 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:56.408061 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:09:56.408407 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.408977 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.409127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:56.409127 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.409127 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:56.409127 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:56.409127 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.409127 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:56.409127 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.409127 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:56.409668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:56.409941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:56.409941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:56.409941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.409941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:56.409941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:56.409941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.409941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:56.409941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.409941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:56.410245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.410245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:56.410245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:56.410245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:56.410245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.410245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:56.410245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:56.410245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:09:56.410245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:56.410245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:56.410245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.410245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:56.410494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:56.410775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.410775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:56.410775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.410775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:56.410913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:56.411243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:56.411573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.411573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:56.411573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:56.411573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.411573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:56.411573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.411573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:56.411732 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:56.411909 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:56.411939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:56.412029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:56.412355 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.412960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.413863 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:56.413863 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.413863 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:56.413863 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:56.413863 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.413863 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:56.413863 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.413863 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:09:56.414401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:09:56.414802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:56.414802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:09:56.414802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.414802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:09:56.414802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:09:56.414802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.414802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:09:56.414802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.414802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:09:56.414984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.414984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:56.414984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:56.414984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:56.414984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.414984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:56.414984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:56.414984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:56.414984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:56.414984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.414984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:56.415205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:56.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:56.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.415623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:56.415929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:56.416297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.416297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:56.416297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.416297 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:09:56.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:09:56.436706 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:09:56.437026 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:09:56.437026 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:09:56.438223 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.438223 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:09:56.438223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.440291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:09:56.440291 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.440291 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:09:56.440291 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:09:56.440291 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.440291 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:09:56.440291 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.440291 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:09:56.440971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.440971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:09:56.440971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:56.440971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:09:56.440971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.440971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:56.440971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:56.440971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.440971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:56.440971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:56.440971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:09:56.440971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.441288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:09:56.441546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.441546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:09:56.442962 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:09:56.443338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:09:56.443701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:09:56.443701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:09:56.443701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:01.078257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:01.078257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:01.078257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:01.078257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:01.118240 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.119029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.119323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:01.119323 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.119323 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:01.119323 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:01.119323 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.119323 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:01.119323 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.119323 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:01.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:01.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:01.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:01.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:01.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:01.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:01.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:01.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:01.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.120655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:01.120880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.120880 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:01.121478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:01.122058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.122503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:01.122503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.122503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:01.399429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:01.399577 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:01.399675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:01.401677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.401677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.401677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.401677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.401677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.401939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.401939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.401939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.401939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.401939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.401939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:01.401939 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.401939 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:01.401939 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.402578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:01.402803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:01.403027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.403027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:01.403027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.403027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:01.403571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:01.403861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.404235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:01.404235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.404235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:01.407944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:01.408170 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:01.408285 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:01.408376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:10:01.409068 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.409068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.409218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:01.409218 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.409218 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:01.409218 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:01.409218 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.409218 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:01.409218 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.409218 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:01.409722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:01.410063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:01.410063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:01.410063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.410063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:01.410063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:01.410063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.410063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:01.410063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.410063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:01.410299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.410299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:01.410299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:01.410299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:01.410299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.410299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:01.410299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:01.410299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:10:01.410299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:01.410299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:01.410299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.410299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:01.410526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:01.410733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.410733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:01.410733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.410733 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:01.410989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:01.411322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:01.411626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.411626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:01.411626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:01.411626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.411626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:01.411626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.411626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:01.411955 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:01.412110 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:01.412205 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:01.412307 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:01.412909 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.412909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.413002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:01.413002 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.413002 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:01.413002 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:01.413002 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.413002 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:01.413002 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.413002 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:01.413514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:01.413755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:01.413755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:01.413755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.413755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:01.413755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:01.413755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.413755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:01.413755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.413755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:01.414076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.414076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:01.414076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:01.414076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:01.414076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.414076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:01.414076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:01.414076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:01.414076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:01.414076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.414076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:01.414346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:01.414564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.414564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:01.414564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.414564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.414781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:01.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:01.415435 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.415435 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:01.415435 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.415435 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:01.436833 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:01.437079 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:01.437255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:01.437423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:01.438503 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.438503 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:01.438503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.438605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:01.438605 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.438605 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:01.438605 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:01.438605 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.438605 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:01.438605 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.438605 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:01.439235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.439235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:01.439235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:01.439235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:01.439235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.439235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:01.439235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:01.439235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.439235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:01.439235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:01.439235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:01.439235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.439623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:01.439923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.439923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:01.440279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:01.440611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:01.440901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:01.440901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:01.440901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:06.076377 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:06.076916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:06.076916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:06.077354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:06.113850 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.114139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.114739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:06.114739 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.114739 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:06.114739 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:06.114739 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.114739 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:06.114739 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.114739 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:06.115517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.115517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:06.115517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:06.115517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:06.115517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.115517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:06.115517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:06.115517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.115517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:06.115517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:06.115517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:06.115517 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.115907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:06.116300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.116300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:06.116649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:06.117114 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.117829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:06.117829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.117829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:06.399853 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:06.400151 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:06.400310 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:06.401322 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.402231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.402231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.402231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.402231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.402231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.402724 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.402724 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.402724 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.402724 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.402724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.403105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:06.403105 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.403105 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:06.403105 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:06.403105 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.403105 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:06.403105 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.403105 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:06.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:06.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:06.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:06.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:06.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:06.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:06.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:06.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:06.403824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:06.404410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.404410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:06.404611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:06.404902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:06.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:06.408979 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:06.411638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:06.411732 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:06.411838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:10:06.412327 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.412570 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:06.412893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.413674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:06.413674 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.413674 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:06.413674 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:06.413674 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.413674 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:06.413674 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.413674 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:06.415411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:06.415704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:06.415704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:06.415704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.415704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:06.415704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:06.415704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.415704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:06.415704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.415704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:06.416029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.416029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:06.416029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:06.416029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:06.416029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.416029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:06.416029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:06.416029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:10:06.416029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:06.416029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:06.416029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.416029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:06.416298 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:06.416568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.416568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:06.416568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.416568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:06.417798 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:06.418344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:06.418932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.418932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:06.418932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:06.418932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.418932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:06.418932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.418932 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:06.418932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:06.418932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:06.418932 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:06.419317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.419317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.419591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:06.419591 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.419591 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:06.419591 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:06.419591 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.419591 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:06.419591 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.419591 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:06.421078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:06.421614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:06.421614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:06.421614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.421614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:06.421614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:06.421614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.421614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:06.421614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.421614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:06.421857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.421857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:06.421857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:06.421857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:06.421857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.421857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:06.421857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:06.421857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:06.421857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:06.421857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.421857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:06.422271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:06.422710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:06.423264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:06.423872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:06.423872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:06.423872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:06.423872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:06.423872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.423872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:06.423872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:06.423872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.423872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:06.423872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.423872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:06.437231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:06.437914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:06.437914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:06.437914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:06.438258 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.438895 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:06.438895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.439937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:06.439937 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.439937 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:06.439937 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:06.439937 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.439937 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:06.439937 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.439937 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:06.441641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.441641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:06.441641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:06.441641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:06.441641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.441641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:06.441641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:06.441641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.441641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:06.441641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:06.441641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:06.441641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.441901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:06.442121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.442121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:06.443049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:06.443365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:06.443665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:06.443665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:06.443665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:11.078475 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:11.080759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:11.080759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:11.080759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:11.112209 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.112440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:11.114615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:11.115218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.115744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:11.116076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:11.116558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:11.116558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:11.116558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.116558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:11.116558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:11.116558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.116558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:11.116558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.116558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:11.400238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:11.400674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:11.400868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:11.402001 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.402232 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.402376 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.402985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.403797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.403797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.403797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.403797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.403797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.403797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.404392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:11.404392 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.404392 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:11.404392 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:11.404392 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.404392 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:11.404392 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.404392 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:11.405385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.405385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:11.405385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:11.405385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:11.405385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.405385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:11.405385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:11.405385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.405385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:11.405385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:11.405385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:11.405385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.408556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:11.409040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:11.409358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.409654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:11.409654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:11.409654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.409654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:11.409654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.409654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:11.411655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:11.411867 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:11.411967 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:11.412075 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:10:11.412390 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.412577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.413025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:11.413025 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.413025 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:11.413025 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:11.413025 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.413025 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:11.413025 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.413025 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:11.414166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:11.415226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:11.416649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:11.418261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:11.418261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:11.418261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:11.418261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:11.418261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:11.418261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.418261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:11.418261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:11.418261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.418261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:11.418261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.418261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:11.419125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:11.420072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:11.421234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.421234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:11.421234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:11.421234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.421234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:11.421234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.421234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:11.421234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:11.421234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:11.421234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:11.421817 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:11.421817 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.421817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.421817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:11.421817 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.421817 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:11.421817 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:11.421817 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.421817 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:11.421817 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.421817 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:11.421817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:11.422544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:11.424348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:11.424583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:11.424784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:11.424784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:11.424784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.424784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:11.424784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:11.424784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.424784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:11.424784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.424784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:11.424784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.424784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:11.424784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.425078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.425410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:11.437699 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:11.438036 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:11.438127 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:11.438513 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:11.438613 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.438869 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:11.439237 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.439940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:11.439940 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.439940 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:11.439940 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:11.439940 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.439940 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:11.439940 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.439940 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:11.440525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.440525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:11.440525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:11.440525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:11.440525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.440525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:11.440525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:11.440525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.440525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:11.440525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:11.440525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:11.440525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.440786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:11.441001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.441001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:11.441650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:11.442285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:11.442613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:11.442613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:11.442613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:16.081459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:16.081916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:16.081916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:16.082485 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:16.121755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.122003 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.122704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:16.122704 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.122704 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:16.122704 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:16.122704 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.122704 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:16.122704 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.122704 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:16.126262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.126262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:16.126262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:16.126262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:16.126262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.126262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:16.126262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:16.126262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.126262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:16.126262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:16.126262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:16.126262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.126505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:16.126717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.126717 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:16.128382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:16.128671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:16.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.128971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:16.400823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:16.401290 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:16.401695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:16.403760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.403935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.404064 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.404161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.404284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.404384 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.404483 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.404583 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.404700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.404803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.404918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.405038 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.405537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:16.405537 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.405537 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:16.405537 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:16.405537 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.405537 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:16.405537 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.405537 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:16.406489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.406489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:16.406489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:16.406489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:16.406489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.406489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:16.406489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:16.406489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.406489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:16.406489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:16.406489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:16.406489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.406814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:16.407062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.407062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:16.407630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:16.408062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.408564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:16.408564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.408564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:16.412107 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:16.412438 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:16.412579 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:16.412717 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:10:16.413055 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.413226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.413656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:16.413656 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.413656 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:16.413656 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:16.413656 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.413656 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:16.413656 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.413656 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:16.414536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:16.414900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:16.414900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:16.414900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.414900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:16.414900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:16.414900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.414900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:16.414900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.414900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:16.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:16.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:16.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:16.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:16.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:16.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:10:16.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:16.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:16.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.415484 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:16.415941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:16.416464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.416464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:16.416464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.416464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:16.416694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:16.417028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:16.417379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.417379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:16.417379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:16.417379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.417379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:16.417379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.417379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:16.419124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:16.419371 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:16.419458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:16.419555 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:16.419856 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.420037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.420425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:16.420425 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.420425 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:16.420425 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:16.420425 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.420425 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:16.420425 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.420425 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:16.420929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:16.421178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:16.421178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:16.421178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.421178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:16.421178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:16.421178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.421178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:16.421178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.421178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:16.421500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.421500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:16.421500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:16.421500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:16.421500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.421500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:16.421500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:16.421500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:16.421500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:16.421500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.421500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:16.421756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:16.422025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.422025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:16.422025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.422025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.422171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:16.422470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:16.422794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.422794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:16.422794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.422794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:16.438113 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:16.438476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:16.438548 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:16.438700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:16.440791 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.440980 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:16.441271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.441785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:16.441785 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.441785 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:16.441785 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:16.441785 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.441785 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:16.441785 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.441785 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:16.442656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.442656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:16.442656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:16.442656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:16.442656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.442656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:16.442656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:16.442656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.442656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:16.442656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:16.442656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:16.442656 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.442927 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:16.443202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.443202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:16.443473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:16.443831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:16.444290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:16.444290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:16.444290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:21.082211 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:21.082211 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:21.082750 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:21.082895 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:21.097210 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=235 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:10:21.128803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.128979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.129443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:21.129443 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.129443 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:21.129443 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:21.129443 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.129443 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:21.129443 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.129443 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:21.130341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.130341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:21.130341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:21.130341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:21.130341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.130341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:21.130341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:21.130341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.130341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:21.130341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:21.130341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:21.130341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.131126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:21.131609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:21.131918 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.132363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:21.132363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:21.132363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.132363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:21.132363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.132363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:21.401261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:21.401674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:21.401798 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:21.402966 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.403147 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.403313 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.403460 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.403605 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.403743 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.403884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.404022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.404168 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.404319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.404453 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.404586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.405067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:21.405067 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.405067 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:21.405067 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:21.405067 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.405067 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:21.405067 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.405067 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:21.405684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.405684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:21.405684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:21.405684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:21.405684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.405684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:21.405684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:21.405684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.405684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:21.405684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:21.405684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:21.405684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.406064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:21.406469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:21.406870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.407433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:21.407433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:21.407433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.407433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:21.407433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.407433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:21.412458 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:21.412776 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:21.412847 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:21.412955 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:10:21.413524 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.413694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.414160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:21.414160 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.414160 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:21.414160 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:21.414160 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.414160 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:21.414160 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.414160 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:21.414760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:21.415277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:21.415277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:21.415277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.415277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:21.415277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:21.415277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.415277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:21.415277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.415277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:21.415642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.415642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:21.415642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:21.415642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:21.415642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.415642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:21.415642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:21.415642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:10:21.415642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:21.415642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:21.415642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.415642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:21.415992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:21.416428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:21.416882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:21.417471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:21.417471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:21.417471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:21.417471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:21.417471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:21.417471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.417471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:21.417471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:21.417471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.417471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:21.417471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.417471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:21.419460 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:21.419763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:21.419842 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:21.419963 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:21.420282 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.420477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.420895 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:21.420895 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.420895 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:21.420895 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:21.420895 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.420895 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:21.420895 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.420895 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:21.421543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:21.422021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:21.422021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:21.422021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.422021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:21.422021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:21.422021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.422021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:21.422021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.422021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:21.422361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.422361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:21.422361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:21.422361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:21.422361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.422361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:21.422361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:21.422361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:21.422361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:21.422361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.422361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:21.422768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:21.423212 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:21.423679 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:21.424371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:21.424371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:21.424371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:21.424371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:21.424371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.424371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:21.424371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:21.424371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.424371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:21.424371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.424371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:21.438670 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:21.438910 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:21.439045 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:21.439280 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:21.441080 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.441292 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:21.441558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.442043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:21.442043 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.442043 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:21.442043 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:21.442043 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.442043 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:21.442043 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.442043 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:21.442715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.442715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:21.442715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:21.442715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:21.442715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.442715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:21.442715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:21.442715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.442715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:21.442715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:21.442715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:21.442715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.443224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:21.443611 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:21.444051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:21.444612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:21.444612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:21.444612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:21.444612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:21.444612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:21.444612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:26.082335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:26.082797 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:26.082899 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:26.083587 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:26.115078 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.115341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.115862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:26.115862 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.115862 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:26.115862 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:26.115862 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.115862 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:26.115862 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.115862 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:26.116543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.116543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:26.116543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:26.116543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:26.116543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.116543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:26.116543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:26.116543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.116543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:26.116543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:26.116543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:26.116543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.116816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:26.117034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.117034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:26.117240 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:26.117536 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.117817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:26.117817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.117817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:26.401685 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:26.401918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:26.402039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:26.405145 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.405362 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.405488 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.405595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.405697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.405800 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.405902 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.405992 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.406086 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.406214 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.406315 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-08c16dd9-f491-44d4-963d-2a87525d5e42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.406416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.406894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:26.406894 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.406894 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:26.406894 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:26.406894 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.406894 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:26.406894 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.406894 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:26.407631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.407631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:26.407631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:26.407631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:26.407631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.407631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:26.407631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:26.407631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.407631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:26.407631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:26.407631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:26.407631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.407878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:26.408132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.408132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:26.408335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:26.408604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:26.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.408930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:26.412702 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:26.412876 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:26.412962 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:26.413075 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:10:26.413423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.413587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.414047 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:26.414047 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.414047 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:26.414047 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:26.414047 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.414047 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:26.414047 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.414047 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:26.414723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:26.415007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:26.415007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:26.415007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.415007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:26.415007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:26.415007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.415007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:26.415007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.415007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:26.415455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.415455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:26.415455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:26.415455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:26.415455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.415455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:26.415455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:26.415455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:10:26.415455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:26.415455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:26.415455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.415455 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:26.415692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:26.415955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.415955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:26.415955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.415955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:26.416329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:26.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:26.416886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.416886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:26.416886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:26.416886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.416886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:26.416886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.416886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:26.419734 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:26.419908 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:26.419996 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:26.420101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:26.420457 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.420694 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.421207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:26.421207 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.421207 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:26.421207 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:26.421207 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.421207 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:26.421207 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.421207 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:26.421718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:26.421985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:26.421985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:26.421985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.421985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:26.421985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:26.421985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.421985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:26.421985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.421985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:26.422391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.422391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:26.422391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:26.422391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:26.422391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.422391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:26.422391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:26.422391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:26.422391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:26.422391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.422391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:26.422583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:26.422843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.422843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:26.422843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.422843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.423115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:26.423436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:26.423742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.423742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:26.423742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.423742 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:26.438836 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:26.439020 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:26.439112 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:26.439248 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:26.439552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.439731 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:26.439969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.440505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:26.440505 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.440505 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:26.440505 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:26.440505 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.440505 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:26.440505 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.440505 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:26.441174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.441174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:26.441174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:26.441174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:26.441174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.441174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:26.441174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:26.441174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.441174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:26.441174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:26.441174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:26.441174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.441434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:26.441684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.441684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:26.442106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:26.442450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:26.442790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:26.442790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:26.442790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:31.083207 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:31.083207 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:31.083986 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:31.084101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:31.116506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.116668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.117058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:31.117058 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.117058 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:31.117058 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:31.117058 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.117058 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:31.117058 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.117058 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:31.117789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.117789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:31.117789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:31.117789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:31.117789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.117789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:31.117789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:31.117789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.117789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:31.117789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:31.117789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:31.117789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.118328 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:31.118843 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:31.119581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.121409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:31.121409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:31.121409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.121409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:31.121409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.121409 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:31.402051 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:31.402326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:31.402446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:31.405200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.405200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-5feabefa-bba2-4b41-a481-a9141d275106, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.405200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.405200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.405200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.405389 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.405389 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.405389 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.405389 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.405389 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.405389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.405515 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:31.405515 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.405515 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:31.405515 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:31.405515 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.405515 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:31.405515 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.405515 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:31.406414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.406414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:31.406414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:31.406414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:31.406414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.406414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:31.406414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:31.406414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.406414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:31.406414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:31.406414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:31.406414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.406651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:31.406847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.406847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:31.407465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:31.407737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.408020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:31.408020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.408020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:31.412950 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:31.413198 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:31.413318 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:31.413437 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:10:31.414134 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.414134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.414306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:31.414306 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.414306 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:31.414306 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:31.414306 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.414306 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:31.414306 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.414306 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:31.415081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:31.415644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:31.415644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:31.415644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.415644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:31.415644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:31.415644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.415644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:31.415644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.415644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:31.415965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.415965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:31.415965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:31.415965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:31.415965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.415965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:31.415965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:31.415965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:10:31.415965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:31.415965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:31.415965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.415965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:31.416487 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:31.416964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.416964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:31.416964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.416964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:31.417096 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:31.417590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:31.418157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.418157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:31.418157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:31.418157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.418157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:31.418157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.418157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:31.419961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:31.420163 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:31.420317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:31.420446 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:31.420859 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.421140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.421676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:31.421676 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.421676 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:31.421676 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:31.421676 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.421676 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:31.421676 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.421676 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:31.422494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:31.423027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:31.423027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:31.423027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.423027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:31.423027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:31.423027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.423027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:31.423027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.423027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:31.423683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.423683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:31.423683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:31.423683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:31.423683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.423683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:31.423683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:31.423683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:31.423683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:31.423683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.423683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:31.424083 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:31.424572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.424572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:31.424572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.424572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.424714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:31.425054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:31.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:31.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:31.439182 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:31.439454 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:31.439578 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:31.439751 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:31.440836 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.440836 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:31.440836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.441419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:31.441419 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.441419 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:31.441419 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:31.441419 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.441419 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:31.441419 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.441419 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:31.441966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.441966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:31.441966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:31.441966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:31.441966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.441966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:31.441966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:31.441966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.441966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:31.441966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:31.441966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:31.441966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.442418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:31.442871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.442871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:31.442940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:31.443548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:31.444159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:31.444159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:31.444159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:36.083311 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:36.083716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:36.083762 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:36.084509 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:36.119949 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.120163 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.120678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:36.120678 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.120678 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:36.120678 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:36.120678 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.120678 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:36.120678 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.120678 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:36.121436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.121436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:36.121436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:36.121436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:36.121436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.121436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:36.121436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:36.121436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.121436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:36.121436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:36.121436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:36.121436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.121749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:36.121998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.121998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:36.122229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:36.122619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.122982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:36.122982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.122982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:36.402513 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:36.402804 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:36.402943 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:36.404057 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.404153 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.404316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.404412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.404513 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.404650 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.404755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.404874 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.404984 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.405122 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.405590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:36.405590 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.405590 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:36.405590 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:36.405590 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.405590 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:36.405590 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.405590 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:36.406441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.406441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:36.406441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:36.406441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:36.406441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.406441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:36.406441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:36.406441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.406441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:36.406441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:36.406441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:36.406441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.406822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:36.407211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.407211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:36.407289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:36.407735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.408294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:36.408294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.408294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:36.413357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:36.413617 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:36.413695 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:36.413818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:10:36.414139 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.414311 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.414687 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:36.414687 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.414687 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:36.414687 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:36.414687 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.414687 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:36.414687 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.414687 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:36.415316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:36.415774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:36.415774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:36.415774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.415774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:36.415774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:36.415774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.415774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:36.415774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.415774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:36.416016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.416016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:36.416016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:36.416016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:36.416016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.416016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:36.416016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:36.416016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:10:36.416016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:36.416016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:36.416016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.416016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:36.416370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:36.416773 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:36.417198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:36.417738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:36.417738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:36.417738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:36.417738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:36.417738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:36.417738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.417738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:36.417738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:36.417738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.417738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:36.417738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.417738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:36.420279 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:36.420520 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:36.420623 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:36.420720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:36.420998 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.421166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.421541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:36.421541 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.421541 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:36.421541 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:36.421541 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.421541 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:36.421541 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.421541 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:36.422108 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:36.422580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:36.422580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:36.422580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.422580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:36.422580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:36.422580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.422580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:36.422580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.422580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:36.422828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.422828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:36.422828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:36.422828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:36.422828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.422828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:36.422828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:36.422828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:36.422828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:36.422828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.422828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:36.423159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:36.423552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:36.423925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:36.424466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:36.424466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:36.424466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:36.424466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:36.424466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.424466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:36.424466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:36.424466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.424466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:36.424466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.424466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:36.442128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:36.442128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:36.442128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:36.442128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:36.442128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.442128 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.442449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:36.442860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:36.443299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:36.443678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:36.444235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:36.444235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:36.444235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:36.444235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:36.444235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:36.444235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:36.444235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:36.444235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:36.444235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:36.444235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:36.444235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:36.444235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:41.083760 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:41.084387 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:41.084387 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:41.084725 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:41.121851 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.121851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.122156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:41.122156 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.122156 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:41.122156 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:41.122156 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.122156 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:41.122156 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.122156 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:41.123006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.123006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:41.123006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:41.123006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:41.123006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.123006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:41.123006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:41.123006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.123006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:41.123006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:41.123006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:41.123006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.123338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:41.123608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.123608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:41.124243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:41.124792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.125471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:41.125471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.125471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:41.402898 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:41.403303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:41.403414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:41.404360 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.405936 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.405936 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.405936 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.405936 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.405936 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.406106 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.406106 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.406106 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.406106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.408043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:41.408043 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.408043 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:41.408043 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:41.408043 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.408043 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:41.408043 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.408043 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:41.408878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.408878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:41.408878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:41.408878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:41.408878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.408878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:41.408878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:41.408878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.408878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:41.408878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:41.408878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:41.408878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.409134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:41.409375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.409375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:41.412095 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:41.412454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.412787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:41.412787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.412787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:41.413756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:41.414033 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:41.414163 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:41.414330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:10:41.415112 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.415112 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.415339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:41.415339 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.415339 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:41.415339 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:41.415339 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.415339 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:41.415339 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.415339 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:41.415975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:41.416343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:41.416343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:41.416343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.416343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:41.416343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:41.416343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.416343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:41.416343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.416343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:41.417528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.417528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:41.417528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:41.417528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:41.417528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.417528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:41.417528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:41.417528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:10:41.417528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:41.417528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:41.417528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.417528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:41.417829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:41.418084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.418084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:41.418084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.418084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:41.420801 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:41.421199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:41.421506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.421506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:41.421506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:41.421506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.421506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:41.421506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.421506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:41.421506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:41.421755 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:41.421914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:41.422076 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:41.422860 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.422860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.423054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:41.423054 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.423054 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:41.423054 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:41.423054 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.423054 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:41.423054 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.423054 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:41.423707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:41.423982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:41.423982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:41.423982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.423982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:41.423982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:41.423982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.423982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:41.423982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.423982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:41.426616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.426616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:41.426616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:41.426616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:41.426616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.426616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:41.426616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:41.426616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:41.426616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:41.426616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.426616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:41.426900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:41.427142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.427142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:41.427142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.427142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.427753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:41.428307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:41.428688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.428688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:41.428688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.428688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:41.440006 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:41.440406 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:41.440524 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:41.440666 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:41.441872 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.441872 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:41.441872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.441972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:41.441972 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.441972 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:41.441972 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:41.441972 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.441972 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:41.441972 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.441972 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:41.443029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.443029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:41.443029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:41.443029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:41.443029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.443029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:41.443029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:41.443029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.443029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:41.443029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:41.443029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:41.443029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.443349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:41.443590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.443590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:41.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:41.444768 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:41.445295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:41.445295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:41.445295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:46.084826 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:46.085383 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:46.085383 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:46.085868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:46.128300 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.128464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.128882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:46.128882 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.128882 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:46.128882 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:46.128882 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.128882 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:46.128882 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.128882 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:46.129590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.129590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:46.129590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:46.129590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:46.129590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.129590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:46.129590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:46.129590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.129590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:46.129590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:46.129590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:46.129590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.129796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:46.130046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.130046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:46.130317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:46.130868 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.131539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:46.131539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.131539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:46.403373 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:46.404255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:46.404255 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:46.404835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.405041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.405209 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.405373 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.405628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.405693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.405831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.405971 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.406127 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.406285 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.406800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:46.406800 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.406800 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:46.406800 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:46.406800 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.406800 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:46.406800 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.406800 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:46.407572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.407572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:46.407572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:46.407572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:46.407572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.407572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:46.407572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:46.407572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.407572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:46.407572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:46.407572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:46.407572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.407833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:46.408043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.408043 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:46.408334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:46.408624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.408955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:46.408955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.408955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:46.414121 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:46.414431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:46.414544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:46.414648 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:10:46.415045 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.415272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.415747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:46.415747 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.415747 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:46.415747 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:46.415747 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.415747 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:46.415747 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.415747 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:46.416594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:46.416906 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:46.417142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:46.417383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:46.417383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:46.417383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:46.417383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:46.417383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:46.417383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.417383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:46.417383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:46.417383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.417383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:46.417383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.417383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:46.417568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:46.417818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:46.418107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.418107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:46.418107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:46.418107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.418107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:46.418107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.418107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:46.421665 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:46.421918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:46.422050 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:46.422252 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:46.422628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.422844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.423323 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:46.423323 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.423323 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:46.423323 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:46.423323 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.423323 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:46.423323 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.423323 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:46.423907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:46.424226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:46.424226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:46.424226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.424226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:46.424226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:46.424226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.424226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:46.424226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.424226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:46.424532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.424532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:46.424532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:46.424532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:46.424532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.424532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:46.424532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:46.424532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:46.424532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:46.424532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.424532 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:46.424800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:46.425068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.425068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:46.425068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.425068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.426032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:46.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:46.426999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.426999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:46.426999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.426999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:46.440368 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:46.440703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:46.440792 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:46.440929 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:46.441610 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.441882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:46.442224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.442771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:46.442771 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.442771 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:46.442771 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:46.442771 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.442771 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:46.442771 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.442771 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:46.444057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.444057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:46.444057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:46.444057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:46.444057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.444057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:46.444057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:46.444057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.444057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:46.444057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:46.444057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:46.444057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.444364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:46.444601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:46.444848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:46.445183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:46.445183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:46.445183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:46.445183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:46.445183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:46.445183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:51.085399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:51.085858 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:51.085858 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:51.086412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:51.112418 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.112566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.112950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:51.112950 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.112950 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:51.112950 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:51.112950 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.112950 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:51.112950 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.112950 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:51.113551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.113551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:51.113551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:51.113551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:51.113551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.113551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:51.113551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:51.113551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.113551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:51.113551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:51.113551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:51.113551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.113839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:51.114106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.114106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:51.114205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:51.114598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.114966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:51.114966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.114966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:51.403901 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:51.404292 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:51.404424 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:51.405738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.405964 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.406129 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.406259 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.406990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.406990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.406990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.406990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.406990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.406990 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.407376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:51.407376 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.407376 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:51.407376 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:51.407376 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.407376 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:51.407376 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.407376 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:51.408019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.408019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:51.408019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:51.408019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:51.408019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.408019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:51.408019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:51.408019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.408019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:51.408019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:51.408019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:51.408019 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.408289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:51.408544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.408544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:51.408784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:51.409075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.409462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:51.409462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.409462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:51.414417 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:51.414650 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:51.414738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:51.414822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:10:51.415079 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.415251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.415588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:51.415588 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.415588 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:51.415588 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:51.415588 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.415588 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:51.415588 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.415588 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:51.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:51.416647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:51.417111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:51.417551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:51.417551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:51.417551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:51.417551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:51.417551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:51.417551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.417551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:51.417551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:51.417551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.417551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:51.417551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.417551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:51.418025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:51.418507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:51.419174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.419174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:51.419174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:51.419174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.419174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:51.419174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.419174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:51.421937 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:51.423091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:51.423201 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:51.423387 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:51.423559 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.423735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.424104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:51.424104 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.424104 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:51.424104 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:51.424104 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.424104 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:51.424104 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.424104 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:51.424620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:51.424886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:51.424886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:51.424886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.424886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:51.424886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:51.424886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.424886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:51.424886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.424886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:51.425227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.425227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:51.425227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:51.425227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:51.425227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.425227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:51.425227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:51.425227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:51.425227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:51.425227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.425227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:51.425432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:51.425663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.425663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:51.425663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.425663 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.425881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:51.426166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:51.426485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.426485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:51.426485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.426485 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:51.440687 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:51.443163 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:51.443272 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:51.443408 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:51.443690 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.443889 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:51.444124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.446576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:51.446576 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.446576 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:51.446576 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:51.446576 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.446576 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:51.446576 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.446576 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:51.447220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.447220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:51.447220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:51.447220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:51.447220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.447220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:51.447220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:51.447220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.447220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:51.447220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:51.447220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:51.447220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.447456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:51.447674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.447674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:51.448203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:51.448521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:51.448857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:51.448857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:51.448857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:56.085765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:56.086282 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:56.086324 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:56.087067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:56.121059 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.121286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.121722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:56.121722 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.121722 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:56.121722 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:56.121722 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.121722 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:56.121722 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.121722 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:56.122403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.122403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:56.122403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:56.122403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:56.122403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.122403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:56.122403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:56.122403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.122403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:56.122403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:56.122403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:56.122403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.122792 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:56.123221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:56.123668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.124241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:56.124241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:56.124241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.124241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:56.124241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.124241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:56.404315 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:56.404708 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:56.404845 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:56.405929 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.406102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.406277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.406422 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.406557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.406696 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.406832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.406971 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.407158 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.407280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.407761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:56.407761 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.407761 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:56.407761 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:56.407761 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.407761 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:56.407761 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.407761 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:56.408750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.408750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:56.408750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:56.408750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:56.408750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.408750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:56.408750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:56.408750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.408750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:56.408750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:56.408750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:56.408750 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.408975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:56.409228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.409228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:56.409967 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:56.410363 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.410686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:56.410686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.410686 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:56.414740 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:56.415012 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:56.415172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:56.415340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:10:56.415626 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.415794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.416205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:56.416205 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.416205 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:56.416205 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:56.416205 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.416205 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:56.416205 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.416205 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:56.417002 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:56.417280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:56.417280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:56.417280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.417280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:56.417280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:56.417280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.417280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:56.417280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.417280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:56.417965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.417965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:56.417965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:56.417965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:56.417965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.417965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:56.417965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:56.417965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:10:56.417965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:56.417965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:56.417965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.417965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:56.418177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:56.418483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.418483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:56.418483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.418483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:56.419113 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:56.419405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:56.419729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.419729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:56.419729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:56.419729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.419729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:56.419729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.419729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:56.423135 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:56.423475 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:56.423603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:56.423756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:56.423992 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.424199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.424595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:56.424595 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.424595 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:56.424595 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:56.424595 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.424595 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:56.424595 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.424595 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:10:56.425436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:10:56.425684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:56.425684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:10:56.425684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.425684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:10:56.425684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:10:56.425684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.425684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:10:56.425684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.425684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:10:56.426450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.426450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:56.426450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:56.426450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:56.426450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.426450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:56.426450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:56.426450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:56.426450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:56.426450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.426450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:56.426664 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:56.426886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.426886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:56.426886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.426886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.427594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:56.427859 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:56.428158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.428158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:56.428158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.428158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:10:56.443289 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:10:56.443606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:10:56.443735 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:10:56.443929 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:10:56.444217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.444400 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:10:56.444615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.445036 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:10:56.445036 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.445036 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:10:56.445036 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:10:56.445036 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.445036 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:10:56.445036 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.445036 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:10:56.445984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.445984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:10:56.445984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:56.445984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:10:56.445984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.445984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:56.445984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:56.445984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.445984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:56.445984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:56.445984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:10:56.445984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.446259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:10:56.446476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.446476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:10:56.447006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:10:56.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:10:56.447695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:10:56.447695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:10:56.447695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:01.086238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:01.088259 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:01.088259 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:01.088259 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:01.116141 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.116419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.116861 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:01.116861 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.116861 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:01.116861 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:01.116861 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.116861 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:01.116861 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.116861 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:01.119116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.119116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:01.119116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:01.119116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:01.119116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.119116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:01.119116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:01.119116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.119116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:01.119116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:01.119116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:01.119116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.119594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:01.120028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.120028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:01.121257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:01.121602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.122143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:01.122143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.122143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:01.404793 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:01.405973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:01.405973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:01.406355 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.406576 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.406762 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.406944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.407143 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.407364 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.407569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.407787 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.407978 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.408208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:01.410221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:01.410588 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.410902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:01.411344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:01.411702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:01.411702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:01.411702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:01.411702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.411702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:01.411702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:01.411702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.411702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:01.411702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.411702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:01.415045 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:01.415913 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:01.415913 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:01.415913 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:11:01.416343 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.416826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.416996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:01.416996 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.416996 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:01.416996 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:01.416996 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.416996 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:01.416996 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.416996 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:01.417762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:01.418221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:01.418221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:01.418221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.418221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:01.418221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:01.418221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.418221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:01.418221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.418221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:01.419904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.419904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:01.419904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:01.419904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:01.419904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.419904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:01.419904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:01.419904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:11:01.419904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:01.419904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:01.419904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.419904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:01.420221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:01.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:01.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:01.421067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:01.421067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:01.421067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:01.421067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:01.421067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:01.421067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.421067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:01.421067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:01.421067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.421067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:01.421067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.421067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:01.424669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:01.424669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:01.424669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:01.424669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:01.424669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.424669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.425012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:01.425012 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.425012 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:01.425012 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:01.425012 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.425012 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:01.425012 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.425012 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:01.425734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:01.426078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:01.426078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:01.426078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.426078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:01.426078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:01.426078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.426078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:01.426078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.426078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:01.427715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.427715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:01.427715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:01.427715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:01.427715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.427715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:01.427715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:01.427715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:01.427715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:01.427715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.427715 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:01.428011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:01.428343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:01.428612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:01.428952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:01.428952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:01.428952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:01.428952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:01.428952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.428952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:01.428952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:01.428952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.428952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:01.428952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.428952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:01.445215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:01.445215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:01.445215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:01.445215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:01.445215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.445215 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:01.445471 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.445760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:01.445760 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.445760 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:01.445760 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:01.445760 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.445760 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:01.445760 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.445760 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:01.446643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.446643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:01.446643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:01.446643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:01.446643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.446643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:01.446643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:01.446643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.446643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:01.446643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:01.446643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:01.446643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.446960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:01.447309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.447309 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:01.448183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:01.448528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:01.448960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:01.448960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:01.448960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:06.088573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:06.088573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:06.088573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:06.088573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:06.120944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.121413 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.122040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:06.122040 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.122040 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:06.122040 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:06.122040 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.122040 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:06.122040 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.122040 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:06.123665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.123665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:06.123665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:06.123665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:06.123665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.123665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:06.123665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:06.123665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.123665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:06.123665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:06.123665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:06.123665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.123891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:06.124094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.124094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:06.124558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:06.125358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.126506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:06.126506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.126506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:06.405420 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:06.406508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:06.406508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:06.406914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.407149 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.407403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.407598 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.407830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.408053 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.408288 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.408554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.408845 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-f1ad9190-c5bc-4ce9-aecd-cecf50430c3d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.409099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.409900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:06.409900 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.409900 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:06.409900 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:06.409900 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.409900 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:06.409900 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.409900 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:06.411040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.411040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:06.411040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:06.411040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:06.411040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.411040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:06.411040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:06.411040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.411040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:06.411040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:06.411040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:06.411040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.412249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:06.412442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:06.412767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.413072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:06.413072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:06.413072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.413072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:06.413072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.413072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:06.415718 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:06.416297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:06.416297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:06.416297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:11:06.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.416915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.417333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:06.417333 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.417333 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:06.417333 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:06.417333 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.417333 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:06.417333 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.417333 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:06.418351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:06.418614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:06.418614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:06.418614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.418614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:06.418614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:06.418614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.418614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:06.418614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.418614 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:06.420205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.420205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:06.420205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:06.420205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:06.420205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.420205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:06.420205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:06.420205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:11:06.420205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:06.420205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:06.420205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.420205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:06.420390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:06.420603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:06.420817 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:06.421105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:06.421105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:06.421105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:06.421105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:06.421105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:06.421105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.421105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:06.421105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:06.421105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.421105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:06.421105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.421105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:06.424756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:06.424756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:06.424756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:06.424756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:06.424756 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.424756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.426028 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:06.426028 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.426028 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:06.426028 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:06.426028 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.426028 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:06.426028 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.426028 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:06.427041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:06.429125 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:06.429541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:06.429851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:06.429851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:06.429851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:06.429851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:06.429851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.429851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:06.429851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:06.429851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.429851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:06.429851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.429851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:06.429851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:06.430734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.431218 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:06.444116 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:06.444400 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:06.444572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:06.444832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:06.445805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.445805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:06.445805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.446032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:06.446032 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.446032 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:06.446032 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:06.446032 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.446032 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:06.446032 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.446032 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:06.446752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.446752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:06.446752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:06.446752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:06.446752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.446752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:06.446752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:06.446752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.446752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:06.446752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:06.446752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:06.446752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.446994 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:06.447264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.447264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:06.447833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:06.448329 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:06.448630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:06.448630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:06.448630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:11.087544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:11.088036 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:11.088094 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:11.088927 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:11.132120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.132354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.132739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:11.132739 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.132739 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:11.132739 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:11.132739 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.132739 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:11.132739 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.132739 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:11.133533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.133533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:11.133533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:11.133533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:11.133533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.133533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:11.133533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:11.133533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.133533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:11.133533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:11.133533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:11.133533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.133949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:11.134360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:11.134819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.135384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:11.135384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:11.135384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.135384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:11.135384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.135384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:11.405868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:11.406136 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:11.406281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:11.406980 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.407160 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.407331 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.407474 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-5e7207fb-ac1d-49ba-b643-8b9b058e6ab1, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.407617 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.407765 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.407905 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.408040 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.408203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.408706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:11.408706 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.408706 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:11.408706 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:11.408706 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.408706 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:11.408706 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.408706 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:11.409438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.409438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:11.409438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:11.409438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:11.409438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.409438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:11.409438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:11.409438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.409438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:11.409438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:11.409438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:11.409438 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.409837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:11.410404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.410404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:11.410601 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:11.410949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.411339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:11.411339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.411339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:11.416015 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:11.416256 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:11.416347 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:11.416439 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:11:11.416621 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.416767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.417093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:11.417093 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.417093 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:11.417093 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:11.417093 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.417093 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:11.417093 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.417093 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:11.417570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:11.417852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:11.417852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:11.417852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.417852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:11.417852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:11.417852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.417852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:11.417852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.417852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:11.418121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.418121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:11.418121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:11.418121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:11.418121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.418121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:11.418121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:11.418121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:11:11.418121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:11.418121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:11.418121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.418121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:11.418391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:11.418629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.418629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:11.418629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.418629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:11.418777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:11.419055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:11.419406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.419406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:11.419406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:11.419406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.419406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:11.419406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.419406 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:11.424019 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:11.424237 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:11.424330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:11.424437 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:11.424639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.424790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.425098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:11.425098 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.425098 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:11.425098 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:11.425098 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.425098 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:11.425098 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.425098 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:11.425589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:11.425848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:11.425848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:11.425848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.425848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:11.425848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:11.425848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.425848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:11.425848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.425848 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:11.426257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.426257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:11.426257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:11.426257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:11.426257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.426257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:11.426257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:11.426257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:11.426257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:11.426257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.426257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:11.426483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:11.426714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.426714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:11.426714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.426714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.426913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:11.427242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:11.427564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.427564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:11.427564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.427564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:11.444540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:11.444772 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:11.444896 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:11.445064 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:11.445350 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.445587 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:11.445849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.446352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:11.446352 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.446352 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:11.446352 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:11.446352 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.446352 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:11.446352 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.446352 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:11.447009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.447009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:11.447009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:11.447009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:11.447009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.447009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:11.447009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:11.447009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.447009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:11.447009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:11.447009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:11.447009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.447453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:11.447889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.447889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:11.448099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:11.448511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:11.448847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:11.448847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:11.448847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:16.089011 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:16.089482 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:16.089524 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:16.090196 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:16.140922 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:16.141138 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.141660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:16.141660 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.141660 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:16.141660 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:16.141660 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.141660 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:16.141660 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.141660 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:16.142426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.052 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.142426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:16.142426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:16.142426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:16.142426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.142426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:16.142426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:16.142426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.142426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:16.142426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:16.142426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:16.142426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.142672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:16.142889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.142889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:16.143147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:16.143488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.143784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:16.143784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.143784 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:16.406335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:16.407678 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:16.407800 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:16.408893 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:16.409074 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:16.409229 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:16.409359 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:16.409471 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:16.409595 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:16.409697 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:16.409805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.410340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:16.410340 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.410340 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:16.410340 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:16.410340 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.410340 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:16.410340 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.410340 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:16.411407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.411407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:16.411407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:16.411407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:16.411407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.411407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:16.411407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:16.411407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.411407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:16.411407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:16.411407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:16.411407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.411669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:16.411925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.411925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:16.412607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:16.412937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.413286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:16.413286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.413286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:16.416349 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:16.416593 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:16.416724 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:16.417139 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:11:16.417602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:16.417772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.418313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:16.418313 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.418313 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:16.418313 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:16.418313 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.418313 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:16.418313 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.418313 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:16.419414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:16.419730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:16.419730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:16.419730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.419730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:16.419730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:16.419730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.419730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:16.419730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.419730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:16.420366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.420366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:16.420366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:16.420366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:16.420366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.420366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:16.420366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:16.420366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:11:16.420366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:16.420366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:16.420366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.420366 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:16.420655 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:16.420888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.420888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:16.420888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.420888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:16.421777 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:16.422068 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:16.422405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.422405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:16.422405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:16.422405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.422405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:16.422405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.422405 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:16.424350 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:16.424593 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:16.424716 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:16.424910 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:16.425357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:16.425573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.426094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:16.426094 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.426094 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:16.426094 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:16.426094 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.426094 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:16.426094 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.426094 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:16.427008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:16.427394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:16.427394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:16.427394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.427394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:16.427394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:16.427394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.427394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:16.427394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.427394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:16.427996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.427996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:16.427996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:16.427996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:16.427996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.427996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:16.427996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:16.427996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:16.427996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:16.427996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.427996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:16.428339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:16.428570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.428570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:16.428570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.428570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.429203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:16.429524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:16.429911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.429911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:16.429911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.429911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:16.444955 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:16.445275 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:16.445407 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:16.445602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:16.446041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:16.446277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:16.446500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.446996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:16.446996 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.446996 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:16.446996 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:16.446996 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.446996 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:16.446996 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.446996 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:16.448063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.448063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:16.448063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:16.448063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:16.448063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.448063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:16.448063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:16.448063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.448063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:16.448063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:16.448063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:16.448063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.448383 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:16.448671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.448671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:16.449353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:16.449672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:16.450026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:16.450026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:16.450026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:21.089425 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:21.089953 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:21.089953 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:21.090725 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:21.103818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=239 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:11:21.134838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:21.134838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.135061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:21.135061 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.135061 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:21.135061 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:21.135061 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.135061 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:21.135061 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.135061 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:21.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:21.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:21.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:21.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:21.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:21.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:21.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:21.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:21.135854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.136161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:21.136441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.136441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:21.136802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:21.137127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.137594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:21.137594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.137594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:21.407684 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:21.407887 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:21.407979 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:21.409669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:21.409669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:21.409669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:21.409669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:21.409669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:21.410208 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:21.410208 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:21.410208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.410208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:21.410208 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.410208 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:21.410208 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:21.410208 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.410208 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:21.410208 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.410208 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:21.410449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.410449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:21.410449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:21.410449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:21.410449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.410449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:21.410449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:21.410449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.410449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:21.410449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:21.410449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:21.410449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.410690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:21.410924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.410924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:21.411247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:21.411584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.411942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:21.411942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.411942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:21.416641 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:21.416808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:21.416886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:21.416969 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:11:21.418575 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:21.418575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.418677 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:21.418677 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.418677 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:21.418677 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:21.418677 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.418677 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:21.418677 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.418677 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:21.419173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:21.419506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:21.419506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:21.419506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.419506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:21.419506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:21.419506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.419506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:21.419506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.419506 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:21.419828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.419828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:21.419828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:21.419828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:21.419828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.419828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:21.419828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:21.419828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:11:21.419828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:21.419828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:21.419828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.419828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:21.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:21.422497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:21.422763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:21.423118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:21.423118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:21.423118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:21.423118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:21.423118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:21.423118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.423118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:21.423118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:21.423118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.423118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:21.423118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.423118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:21.424585 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:21.424737 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:21.424817 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:21.424909 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:21.425173 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:21.425444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.426392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:21.426392 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.426392 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:21.426392 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:21.426392 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.426392 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:21.426392 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.426392 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:21.426929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:21.427241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:21.427241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:21.427241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.427241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:21.427241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:21.427241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.427241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:21.427241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.427241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:21.427599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.427599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:21.427599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:21.427599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:21.427599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.427599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:21.427599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:21.427599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:21.427599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:21.427599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.427599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:21.427846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:21.428134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.428134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:21.428134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.428134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.428325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:21.428631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:21.428960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.428960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:21.428960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.428960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:21.445269 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:21.445437 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:21.445518 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:21.445659 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:21.446613 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:21.446613 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:21.446613 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.446753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:21.446753 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.446753 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:21.446753 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:21.446753 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.446753 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:21.446753 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.446753 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:21.447254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.447254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:21.447254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:21.447254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:21.447254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.447254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:21.447254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:21.447254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.447254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:21.447254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:21.447254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:21.447254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.447494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:21.447729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.447729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:21.447905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:21.448234 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:21.448599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:21.448599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:21.448599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:24.159063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:24.159445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:24.159445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:24.166035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 09 01:11:24.166178 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Number of inconsistencies found at create/update: networks=9, security_groups=2, routers=6, router_ports=6 {{(pid=89255) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 09 01:11:24.166292 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=12 {{(pid=89255) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 09 01:11:24.166381 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource b3e28628-ad68-4ca3-b181-b39c995a69e0 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.192930 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, resource={'id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'name': 'public', 'tenant_id': '5a148cfd27844ee6ad0d79cb077e4985', 'project_id': '5a148cfd27844ee6ad0d79cb077e4985', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['17135a44-456c-43be-8f7d-7714aa3731cf', 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-09T00:51:13Z', 'updated_at': '2026-06-09T00:51:20Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.193231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.193597 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-6a91ff68-89ba-411b-b54f-ff58910954a3, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.193832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.194427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.194427 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.194427 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.194427 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.194427 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.194427 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.194427 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.194427 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.195358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource b3e28628-ad68-4ca3-b181-b39c995a69e0 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.195358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.195358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.195358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.195358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.195358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.195358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:11:24.195358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.195358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.195358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.195358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.195627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.195846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.195846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.195846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.195846 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource fe4ad12a-a418-4360-a09f-07d6cf05bf34 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.220301 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-fe4ad12a-a418-4360-a09f-07d6cf05bf34, resource={'id': 'fe4ad12a-a418-4360-a09f-07d6cf05bf34', 'name': 'tempest-TestEncryptedCinderVolumes-657626301-network', 'tenant_id': '6be708d78888439dae8b5dd7299622cf', 'project_id': '6be708d78888439dae8b5dd7299622cf', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['f8f23a40-e583-440a-b23b-0d6c74d9d2b3'], 'standard_attr_id': 1647, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:10:04Z', 'updated_at': '2026-06-09T01:10:04Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 44107}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.220474 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-fe4ad12a-a418-4360-a09f-07d6cf05bf34, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-657626301-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.220681 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-b024ab0e-78cd-4354-9866-f844f1cbb788, external_ids_update=None, columns={'tag': 44107}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.220770 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.221183 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.222024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource fe4ad12a-a418-4360-a09f-07d6cf05bf34 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.222024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.222024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.222024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.222024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.222024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.222024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:11:24.222024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.222024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.222024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.222024 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.222271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.222604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.222604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.222604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.222604 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource a3ce1939-fc7a-4e92-b53c-c522f498b9fd (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.246434 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-a3ce1939-fc7a-4e92-b53c-c522f498b9fd, resource={'id': 'a3ce1939-fc7a-4e92-b53c-c522f498b9fd', 'name': 'tempest-TestVolumeBootPatternV346-746043852-network', 'tenant_id': 'cdd8228d6b2a49f8b9759731d996748b', 'project_id': 'cdd8228d6b2a49f8b9759731d996748b', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['8c92e592-5643-4be1-84e5-a963116f4681'], 'standard_attr_id': 1559, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:08:26Z', 'updated_at': '2026-06-09T01:08:26Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 17628}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.246700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-a3ce1939-fc7a-4e92-b53c-c522f498b9fd, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPatternV346-746043852-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.247025 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-c06b8a92-f9e0-4952-8062-a3cfff369d30, external_ids_update=None, columns={'tag': 17628}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.247164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.247722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.247722 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.247722 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.247722 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.247722 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.247722 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.247722 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.247722 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.248591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource a3ce1939-fc7a-4e92-b53c-c522f498b9fd (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.248591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.248591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.248591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.248591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.248591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.248591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:11:24.248591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.248591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.248591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.248591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.248971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.249392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.249392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.249392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.249392 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 3110b535-aaec-4a8b-800a-0c134bfcd8c9 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.278010 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-3110b535-aaec-4a8b-800a-0c134bfcd8c9, resource={'id': '3110b535-aaec-4a8b-800a-0c134bfcd8c9', 'name': 'tempest-TestEncryptedCinderVolumes-1577079828-network', 'tenant_id': '74bb61daa4fd4148954eef29caf503ee', 'project_id': '74bb61daa4fd4148954eef29caf503ee', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['35d89ff1-84b9-475b-b2fa-548d113e6d9e'], 'standard_attr_id': 1659, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:10:09Z', 'updated_at': '2026-06-09T01:10:09Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 48506}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.278423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-3110b535-aaec-4a8b-800a-0c134bfcd8c9, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-1577079828-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.278504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-73138c84-a25c-497a-9e8c-034f694df855, external_ids_update=None, columns={'tag': 48506}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.278597 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.279124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.279124 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.279124 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.279124 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.279124 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.279124 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.279124 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.279124 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.280267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 3110b535-aaec-4a8b-800a-0c134bfcd8c9 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.280267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.280267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.280267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.280267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.280267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.280267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:11:24.280267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.280267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.280267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.280267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.280544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.280766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.280766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.280766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.280766 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 792be9e1-c9fb-40f3-aaf6-8aa81b356255 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.307287 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-792be9e1-c9fb-40f3-aaf6-8aa81b356255, resource={'id': '792be9e1-c9fb-40f3-aaf6-8aa81b356255', 'name': 'tempest-TestVolumeBootPatternV346-2086662196-network', 'tenant_id': '8665f53d42a14b0d972999e9fa29004f', 'project_id': '8665f53d42a14b0d972999e9fa29004f', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['6f82108b-23d7-4c73-90f8-cba5a1814f1f'], 'standard_attr_id': 1540, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:08:21Z', 'updated_at': '2026-06-09T01:08:21Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 41806}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.307498 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-792be9e1-c9fb-40f3-aaf6-8aa81b356255, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPatternV346-2086662196-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.307777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-2c809ffe-e99e-4c6a-aa67-02de3caac0fe, external_ids_update=None, columns={'tag': 41806}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.307877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.308448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.308448 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.308448 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.308448 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.308448 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.308448 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.308448 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.308448 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.309673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 792be9e1-c9fb-40f3-aaf6-8aa81b356255 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.309673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.309673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.309673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.309673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.309673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.309673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:11:24.309673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.309673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.309673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.309673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.309881 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.310105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.310105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.310105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.310105 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource b5d0a744-9cc6-42b7-9918-c05780a6b728 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.336544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-b5d0a744-9cc6-42b7-9918-c05780a6b728, resource={'id': 'b5d0a744-9cc6-42b7-9918-c05780a6b728', 'name': 'tempest-TestEncryptedCinderVolumes-2075718584-network', 'tenant_id': 'ee7aa68340384985879f1087638be6c4', 'project_id': 'ee7aa68340384985879f1087638be6c4', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['7e4ca6cd-dfd0-4887-8853-d64ab4d2aabf'], 'standard_attr_id': 1436, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:06:00Z', 'updated_at': '2026-06-09T01:06:00Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 64547}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.336725 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-b5d0a744-9cc6-42b7-9918-c05780a6b728, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-2075718584-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.337028 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-ce3fc72d-eb07-4966-90cc-d674aa2d9247, external_ids_update=None, columns={'tag': 64547}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.337175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.337626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.337626 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.337626 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.337626 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.337626 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.337626 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.337626 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.337626 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.338583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource b5d0a744-9cc6-42b7-9918-c05780a6b728 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.338583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.338583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.338583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.338583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.338583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.338583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:11:24.338583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.338583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.338583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.338583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.338938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.339528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.339528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.339528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.339528 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource dc15bda6-1b51-4a9e-af84-ac570d199c11 (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.366151 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-dc15bda6-1b51-4a9e-af84-ac570d199c11, resource={'id': 'dc15bda6-1b51-4a9e-af84-ac570d199c11', 'name': 'private', 'tenant_id': '5a148cfd27844ee6ad0d79cb077e4985', 'project_id': '5a148cfd27844ee6ad0d79cb077e4985', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['65e82083-c074-4bc8-b13e-2eeabecc299d', '6f142de2-8acf-46b2-af54-c34cf691eff5'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:51:08Z', 'updated_at': '2026-06-09T00:51:11Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 62028}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.366478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-dc15bda6-1b51-4a9e-af84-ac570d199c11, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.366831 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-77925902-9e1b-4f7c-868e-70a5d73ee785, external_ids_update=None, columns={'tag': 62028}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.366951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.367925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.367925 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.367925 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.367925 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.367925 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.367925 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.367925 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.367925 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.368429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource dc15bda6-1b51-4a9e-af84-ac570d199c11 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.368429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.368429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.368429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.368429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.368429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.368429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:11:24.368429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.368429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.368429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.368429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.368739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.368974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.368974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.368974 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.368974 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 65be45e6-6fc2-4d7e-a11f-9c52a7128c5e (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.393705 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-65be45e6-6fc2-4d7e-a11f-9c52a7128c5e, resource={'id': '65be45e6-6fc2-4d7e-a11f-9c52a7128c5e', 'name': 'tempest-TestEncryptedCinderVolumes-21700818-network', 'tenant_id': 'c87add1c34bc436bb9af4621f2754703', 'project_id': 'c87add1c34bc436bb9af4621f2754703', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['794da3e4-5cb6-4717-8f32-688b4dc675bf'], 'standard_attr_id': 1423, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T01:05:54Z', 'updated_at': '2026-06-09T01:05:55Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 9995}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.393933 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-65be45e6-6fc2-4d7e-a11f-9c52a7128c5e, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-21700818-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.394156 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-7bff860f-8c74-434d-9827-4c38fb7797bf, external_ids_update=None, columns={'tag': 9995}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.394260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.394778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.394778 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.394778 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.394778 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.394778 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.394778 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.394778 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.394778 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.395629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 65be45e6-6fc2-4d7e-a11f-9c52a7128c5e (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.395629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.395629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.395629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.395629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.395629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.395629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:11:24.395629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.395629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.395629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.395629 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.395860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.396238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.396238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.396238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.396238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 05419c49-9db5-4d5d-8bbe-5a7016c12d7e (type: networks) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.420411 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-05419c49-9db5-4d5d-8bbe-5a7016c12d7e, resource={'id': '05419c49-9db5-4d5d-8bbe-5a7016c12d7e', 'name': 'shared', 'tenant_id': '5a148cfd27844ee6ad0d79cb077e4985', 'project_id': '5a148cfd27844ee6ad0d79cb077e4985', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['612c5f15-747e-4ea8-a2d1-d0bbb8f1bc11'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-09T00:52:24Z', 'updated_at': '2026-06-09T00:52:25Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 44634}, resource_type=networks, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.420710 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-05419c49-9db5-4d5d-8bbe-5a7016c12d7e, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.421024 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-1c288345-932b-445f-8f63-edb47405877c, external_ids_update=None, columns={'tag': 44634}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.421172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.422428 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.422428 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.422428 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.422428 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.422428 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.422428 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.422428 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.422428 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.423277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 05419c49-9db5-4d5d-8bbe-5a7016c12d7e (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.423277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.423277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.423277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.423277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.423277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.423277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2366, in update_network Jun 09 01:11:24.423277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.423277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.423277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.423277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.423642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.424044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.424044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.424044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.424044 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource cdc9f73c-4de3-4f97-9e7b-c61f8fa221dd (type: security_groups) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.446669 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Successfully bumped revision number for resource cdc9f73c-4de3-4f97-9e7b-c61f8fa221dd (type: security_groups) to 3 Jun 09 01:11:24.446793 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 23824127-04fa-47b7-b9bd-38f3e26b6432 (type: security_groups) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.466971 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Successfully bumped revision number for resource 23824127-04fa-47b7-b9bd-38f3e26b6432 (type: security_groups) to 3 Jun 09 01:11:24.467085 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 2ce13c62-602b-4159-a60a-7c5b51ca01e4 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.532173 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, resource={'id': '2ce13c62-602b-4159-a60a-7c5b51ca01e4', 'name': 'tempest-TestEncryptedCinderVolumes-2075718584-router', 'project_id': 'ee7aa68340384985879f1087638be6c4', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.241'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::10'}], 'enable_snat': True}, 'gw_port_id': '1f0502fd-5275-4f42-93db-f6042f6f1fe0', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.241', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::10', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:06:01Z', 'updated_at': '2026-06-09T01:06:03Z', 'revision_number': 3, 'tenant_id': 'ee7aa68340384985879f1087638be6c4'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.532497 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-2075718584-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.532642 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "1f0502fd-5275-4f42-93db-f6042f6f1fe0"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.532642 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "1f0502fd-5275-4f42-93db-f6042f6f1fe0"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.532788 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.533382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.533382 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.533382 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.533382 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.533382 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.533382 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.533382 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.533382 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.533382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 2ce13c62-602b-4159-a60a-7c5b51ca01e4. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.534416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 2ce13c62-602b-4159-a60a-7c5b51ca01e4 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.534416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.534416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.534416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.534416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.534416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.534416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:11:24.534416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:11:24.534416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:11:24.534416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:11:24.534416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.535070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.535533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.535533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.535533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.535533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.535533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.535533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.535533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.535533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.535533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.535533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource d2699db0-e9de-4cd3-b18f-b5c822635662 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.598354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, resource={'id': 'd2699db0-e9de-4cd3-b18f-b5c822635662', 'name': 'tempest-TestEncryptedCinderVolumes-1577079828-router', 'project_id': '74bb61daa4fd4148954eef29caf503ee', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.134'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::1a'}], 'enable_snat': True}, 'gw_port_id': '23b57400-a8ed-4a26-a9c6-de0129713675', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.134', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::1a', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:10:10Z', 'updated_at': '2026-06-09T01:10:12Z', 'revision_number': 3, 'tenant_id': '74bb61daa4fd4148954eef29caf503ee'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.598655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-1577079828-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.598729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "23b57400-a8ed-4a26-a9c6-de0129713675"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.598966 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "23b57400-a8ed-4a26-a9c6-de0129713675"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.598966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.599598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.599598 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.599598 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.599598 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.599598 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.599598 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.599598 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.599598 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.599799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router d2699db0-e9de-4cd3-b18f-b5c822635662. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.600680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource d2699db0-e9de-4cd3-b18f-b5c822635662 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.600680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.600680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.600680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.600680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.600680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.600680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:11:24.600680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:11:24.600680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:11:24.600680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:11:24.600680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.600921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.601166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.601166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.601166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.601166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.601166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.601166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.601166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.601166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.601166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.601166 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource ebe02836-594c-4689-86a7-929dc6b874e7 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.662587 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, resource={'id': 'ebe02836-594c-4689-86a7-929dc6b874e7', 'name': 'tempest-TestEncryptedCinderVolumes-21700818-router', 'project_id': 'c87add1c34bc436bb9af4621f2754703', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.195'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::39c'}], 'enable_snat': True}, 'gw_port_id': 'b7fb27fe-d2c2-4a0d-b76b-78871323a7b4', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.195', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::39c', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:05:55Z', 'updated_at': '2026-06-09T01:05:58Z', 'revision_number': 3, 'tenant_id': 'c87add1c34bc436bb9af4621f2754703'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.662803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-21700818-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.662966 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "b7fb27fe-d2c2-4a0d-b76b-78871323a7b4"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.663091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "b7fb27fe-d2c2-4a0d-b76b-78871323a7b4"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.663259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.663760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.663760 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.663760 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.663760 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.663760 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.663760 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.663760 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.663760 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.663939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router ebe02836-594c-4689-86a7-929dc6b874e7. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.665050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource ebe02836-594c-4689-86a7-929dc6b874e7 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.665050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.665050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.665050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.665050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.665050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.665050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:11:24.665050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:11:24.665050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:11:24.665050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:11:24.665050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.665318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.665721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.665721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.665721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.665721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.665721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.665721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.665721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.665721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.665721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.665721 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 8b5061d7-f426-4fdf-8094-997a01b6252a (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.724262 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, resource={'id': '8b5061d7-f426-4fdf-8094-997a01b6252a', 'name': 'tempest-TestVolumeBootPatternV346-2086662196-router', 'project_id': '8665f53d42a14b0d972999e9fa29004f', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.174'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::2fd'}], 'enable_snat': True}, 'gw_port_id': '22ce133c-c526-4b7f-a60b-fe8072b31672', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.174', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::2fd', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:08:22Z', 'updated_at': '2026-06-09T01:08:24Z', 'revision_number': 3, 'tenant_id': '8665f53d42a14b0d972999e9fa29004f'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.724465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPatternV346-2086662196-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.724465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "22ce133c-c526-4b7f-a60b-fe8072b31672"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.724465 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "22ce133c-c526-4b7f-a60b-fe8072b31672"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.724465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.724465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.724465 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.724465 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.724465 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.724704 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.724704 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.724704 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.724704 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.724704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 8b5061d7-f426-4fdf-8094-997a01b6252a. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.725449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 8b5061d7-f426-4fdf-8094-997a01b6252a (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.725449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.725449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.725449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.725449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.725449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.725449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:11:24.725449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:11:24.725449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:11:24.725449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:11:24.725449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.725844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.726260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.726260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.726260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.726260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.726260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.726260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.726260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.726260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.726260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.726260 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 2168c57b-e149-43e1-b184-bf21d9c205cd (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.796028 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, resource={'id': '2168c57b-e149-43e1-b184-bf21d9c205cd', 'name': 'tempest-TestVolumeBootPatternV346-746043852-router', 'project_id': 'cdd8228d6b2a49f8b9759731d996748b', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.230'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::199'}], 'enable_snat': True}, 'gw_port_id': 'e849c6b9-b88f-4f6f-9993-5c5a9f0f548d', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.230', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::199', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:08:27Z', 'updated_at': '2026-06-09T01:08:28Z', 'revision_number': 3, 'tenant_id': 'cdd8228d6b2a49f8b9759731d996748b'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.796524 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPatternV346-746043852-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.796882 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "e849c6b9-b88f-4f6f-9993-5c5a9f0f548d"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.797174 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "e849c6b9-b88f-4f6f-9993-5c5a9f0f548d"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.797452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.798154 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.798154 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.798154 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.798154 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.798154 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.798154 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.798154 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.798154 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.798314 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 2168c57b-e149-43e1-b184-bf21d9c205cd. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.799307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 2168c57b-e149-43e1-b184-bf21d9c205cd (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.799307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.799307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.799307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.799307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.799307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.799307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:11:24.799307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:11:24.799307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:11:24.799307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:11:24.799307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.799519 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.799779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.799779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.799779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.799779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.799779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.799779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.799779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.799779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.799779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.799779 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 30d72b29-5501-41ca-ae9f-1cdc7119aa42 (type: routers) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.857707 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, resource={'id': '30d72b29-5501-41ca-ae9f-1cdc7119aa42', 'name': 'tempest-TestEncryptedCinderVolumes-657626301-router', 'project_id': '6be708d78888439dae8b5dd7299622cf', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.128'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::18f'}], 'enable_snat': True}, 'gw_port_id': 'e4597a2f-b49c-4d08-a146-c232cf404f7e', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'external_fixed_ips': [{'ip_address': '172.24.5.128', 'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b'}, {'ip_address': '2001:db8::18f', 'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-09T01:10:05Z', 'updated_at': '2026-06-09T01:10:07Z', 'revision_number': 3, 'tenant_id': '6be708d78888439dae8b5dd7299622cf'}, resource_type=routers, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.858045 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-657626301-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.858317 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "e4597a2f-b49c-4d08-a146-c232cf404f7e"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.858489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "e4597a2f-b49c-4d08-a146-c232cf404f7e"')], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.858622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.859356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.859356 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.859356 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.859356 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.859356 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.859356 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.859356 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.859356 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.859514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Unable to update router 30d72b29-5501-41ca-ae9f-1cdc7119aa42. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.860489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 30d72b29-5501-41ca-ae9f-1cdc7119aa42 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.860489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.860489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.860489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.860489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.860489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.860489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1664, in update_router Jun 09 01:11:24.860489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 09 01:11:24.860489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 09 01:11:24.860489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 09 01:11:24.860489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1598, in update_router Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.860724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.860953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.860953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.860953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.860953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.860953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.860953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.860953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.860953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.860953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.860953 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource e4597a2f-b49c-4d08-a146-c232cf404f7e (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:24.933738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-e4597a2f-b49c-4d08-a146-c232cf404f7e, resource={'id': 'e4597a2f-b49c-4d08-a146-c232cf404f7e', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:08:30:89', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '30d72b29-5501-41ca-ae9f-1cdc7119aa42', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1652, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.128'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::18f'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:10:05Z', 'updated_at': '2026-06-09T01:10:07Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.934036 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-e4597a2f-b49c-4d08-a146-c232cf404f7e, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::18f/64', '172.24.5.128/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.934468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=e4597a2f-b49c-4d08-a146-c232cf404f7e, lrouter_port=lrp-e4597a2f-b49c-4d08-a146-c232cf404f7e, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:24.934738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.935382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:24.935382 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.935382 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:24.935382 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:24.935382 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.935382 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:24.935382 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.935382 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:24.936153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource e4597a2f-b49c-4d08-a146-c232cf404f7e (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.936153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:24.936153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:24.936153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:24.936153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:24.936153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:24.936153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:11:24.936153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:24.936153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.936153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.936153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:24.936403 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:24.936649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:24.936649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:24.936649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:24.936649 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 23b57400-a8ed-4a26-a9c6-de0129713675 (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:25.035700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-23b57400-a8ed-4a26-a9c6-de0129713675, resource={'id': '23b57400-a8ed-4a26-a9c6-de0129713675', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:b1:4f:91', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'd2699db0-e9de-4cd3-b18f-b5c822635662', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1664, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.134'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::1a'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:10:10Z', 'updated_at': '2026-06-09T01:10:14Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.035990 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-23b57400-a8ed-4a26-a9c6-de0129713675, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-d2699db0-e9de-4cd3-b18f-b5c822635662'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::1a/64', '172.24.5.134/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.036357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=23b57400-a8ed-4a26-a9c6-de0129713675, lrouter_port=lrp-23b57400-a8ed-4a26-a9c6-de0129713675, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.036589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.037115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:25.037115 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:25.037115 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:25.037115 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:25.037115 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:25.037115 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:25.037115 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.037115 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:25.037885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 23b57400-a8ed-4a26-a9c6-de0129713675 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.037885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:25.037885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:25.037885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:25.037885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:25.037885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:25.037885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:11:25.037885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:25.037885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:25.037885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:25.037885 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:25.038089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:25.038344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:25.038344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.038344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:25.038344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource e849c6b9-b88f-4f6f-9993-5c5a9f0f548d (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:25.123612 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-e849c6b9-b88f-4f6f-9993-5c5a9f0f548d, resource={'id': 'e849c6b9-b88f-4f6f-9993-5c5a9f0f548d', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:13:d4:62', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '2168c57b-e149-43e1-b184-bf21d9c205cd', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1565, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.230'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::199'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:08:27Z', 'updated_at': '2026-06-09T01:08:28Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.124098 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-e849c6b9-b88f-4f6f-9993-5c5a9f0f548d, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-2168c57b-e149-43e1-b184-bf21d9c205cd'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.230/24', '2001:db8::199/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.125411 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=e849c6b9-b88f-4f6f-9993-5c5a9f0f548d, lrouter_port=lrp-e849c6b9-b88f-4f6f-9993-5c5a9f0f548d, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.125411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.125411 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:25.125411 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:25.125411 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:25.125411 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:25.125411 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:25.125411 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:25.125411 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.125411 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:25.125893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource e849c6b9-b88f-4f6f-9993-5c5a9f0f548d (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.125893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:25.125893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:25.125893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:25.125893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:25.125893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:25.125893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:11:25.125893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:25.125893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:25.125893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:25.125893 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:25.126121 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:25.126392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:25.126392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.126392 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:25.126392 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 22ce133c-c526-4b7f-a60b-fe8072b31672 (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:25.203967 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-22ce133c-c526-4b7f-a60b-fe8072b31672, resource={'id': '22ce133c-c526-4b7f-a60b-fe8072b31672', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:c1:2a:8d', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '8b5061d7-f426-4fdf-8094-997a01b6252a', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1545, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.174'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::2fd'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:08:22Z', 'updated_at': '2026-06-09T01:08:24Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.205512 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-22ce133c-c526-4b7f-a60b-fe8072b31672, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-8b5061d7-f426-4fdf-8094-997a01b6252a'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::2fd/64', '172.24.5.174/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.205854 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=22ce133c-c526-4b7f-a60b-fe8072b31672, lrouter_port=lrp-22ce133c-c526-4b7f-a60b-fe8072b31672, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.206321 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.207890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:25.207890 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:25.207890 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:25.207890 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:25.207890 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:25.207890 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:25.207890 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.207890 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:25.208775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 22ce133c-c526-4b7f-a60b-fe8072b31672 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.208775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:25.208775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:25.208775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:25.208775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:25.208775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:25.208775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:11:25.208775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:25.208775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:25.208775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:25.208775 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:25.209176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:25.209579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:25.209579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.209579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:25.209579 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 1f0502fd-5275-4f42-93db-f6042f6f1fe0 (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:25.305120 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-1f0502fd-5275-4f42-93db-f6042f6f1fe0, resource={'id': '1f0502fd-5275-4f42-93db-f6042f6f1fe0', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:8c:91:cc', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '2ce13c62-602b-4159-a60a-7c5b51ca01e4', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1441, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.241'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::10'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:06:01Z', 'updated_at': '2026-06-09T01:06:03Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.305677 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-1f0502fd-5275-4f42-93db-f6042f6f1fe0, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.241/24', '2001:db8::10/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.306161 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=1f0502fd-5275-4f42-93db-f6042f6f1fe0, lrouter_port=lrp-1f0502fd-5275-4f42-93db-f6042f6f1fe0, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.306574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.307349 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:25.307349 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:25.307349 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:25.307349 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:25.307349 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:25.307349 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:25.307349 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.307349 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:25.308256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource 1f0502fd-5275-4f42-93db-f6042f6f1fe0 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.308256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:25.308256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:25.308256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:25.308256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:25.308256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:25.308256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:11:25.308256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:25.308256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:25.308256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:25.308256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:25.308465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:25.308680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:25.308680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.308680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:25.308680 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource b7fb27fe-d2c2-4a0d-b76b-78871323a7b4 (type: router_ports) at create/update {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 09 01:11:25.395701 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-b7fb27fe-d2c2-4a0d-b76b-78871323a7b4, resource={'id': 'b7fb27fe-d2c2-4a0d-b76b-78871323a7b4', 'name': '', 'network_id': 'b3e28628-ad68-4ca3-b181-b39c995a69e0', 'project_id': '', 'mac_address': 'fa:16:3e:83:03:af', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'ebe02836-594c-4689-86a7-929dc6b874e7', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1428, 'fixed_ips': [{'subnet_id': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b', 'ip_address': '172.24.5.195'}, {'subnet_id': '17135a44-456c-43be-8f7d-7714aa3731cf', 'ip_address': '2001:db8::39c'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000002381', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-09T01:05:55Z', 'updated_at': '2026-06-09T01:05:58Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.396241 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-b7fb27fe-d2c2-4a0d-b76b-78871323a7b4, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': 'c7b16d21-e944-4fbd-ba88-8ca4031ed08b 17135a44-456c-43be-8f7d-7714aa3731cf', 'neutron:network_name': 'neutron-b3e28628-ad68-4ca3-b181-b39c995a69e0', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-ebe02836-594c-4689-86a7-929dc6b874e7'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.195/24', '2001:db8::39c/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.396604 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=b7fb27fe-d2c2-4a0d-b76b-78871323a7b4, lrouter_port=lrp-b7fb27fe-d2c2-4a0d-b76b-78871323a7b4, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:25.396834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.397486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:25.397486 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:25.397486 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:25.397486 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:25.397486 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:25.397486 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:25.397486 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.397486 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:25.398229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Failed to fix resource b7fb27fe-d2c2-4a0d-b76b-78871323a7b4 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.398229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:25.398229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 09 01:11:25.398229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 09 01:11:25.398229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 09 01:11:25.398229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 09 01:11:25.398229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2005, in update_router_port Jun 09 01:11:25.398229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:25.398229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:25.398229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:25.398229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:25.398458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:25.398683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:25.398683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:25.398683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:25.398683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 23f291dc-efcc-4517-a355-ee38f017fdde (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:11:25.398683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(23f291dc-efcc-4517-a355-ee38f017fdde) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:11:25.403998 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 2b38ec0a-d126-4884-a204-bc54bdc89034 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:11:25.404328 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(2b38ec0a-d126-4884-a204-bc54bdc89034) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:11:25.409945 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 445ba689-0dfb-43fd-8c3b-14d0c9227389 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:11:25.410220 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(445ba689-0dfb-43fd-8c3b-14d0c9227389) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:11:25.414328 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 4abb940c-0e56-4258-9c51-4bebfb8b2751 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:11:25.414556 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(4abb940c-0e56-4258-9c51-4bebfb8b2751) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:11:25.418942 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 4cef3b97-e0f2-4bb0-ad37-19f677f1f6ef (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:11:25.419169 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(4cef3b97-e0f2-4bb0-ad37-19f677f1f6ef) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:11:25.423038 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 6e0cd7e9-ac47-4dfb-93fb-ccb2de321bb9 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:11:25.423335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(6e0cd7e9-ac47-4dfb-93fb-ccb2de321bb9) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:11:25.427820 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 88aa1dbf-106a-445b-8e36-c1bd3858be24 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:11:25.428094 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(88aa1dbf-106a-445b-8e36-c1bd3858be24) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:11:25.431795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource 918511a3-ffa0-4e0c-ab1b-367d461fbb3d (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:11:25.432037 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(918511a3-ffa0-4e0c-ab1b-367d461fbb3d) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:11:25.436352 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource a42705d4-4307-4f1c-b570-cf1a425ca43d (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:11:25.436616 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(a42705d4-4307-4f1c-b570-cf1a425ca43d) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:11:25.440833 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource a7b16dc5-6418-4259-906a-6631214dd6e1 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:11:25.441782 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(a7b16dc5-6418-4259-906a-6631214dd6e1) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:11:25.446586 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource b2eaed9d-6a07-480b-9aa7-4db921328166 (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:11:25.447326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(b2eaed9d-6a07-480b-9aa7-4db921328166) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:11:25.451965 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Maintenance task: Fixing resource bb3d4216-6a56-4092-9ac7-8a89e97dc07f (type: router_ports) at delete {{(pid=89255) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 09 01:11:25.452259 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.db.ovn_revision_numbers_db [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] delete_revision(bb3d4216-6a56-4092-9ac7-8a89e97dc07f) {{(pid=89255) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 09 01:11:25.457505 np0000002381 neutron-ovn-maintenance-worker[89255]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_for_inconsistencies finished in 1.298 seconds Jun 09 01:11:26.090033 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:26.090639 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:26.090728 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:26.091382 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:26.119988 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:26.120358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.120969 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:26.120969 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.120969 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:26.120969 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:26.120969 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.120969 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:26.120969 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.120969 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:26.121683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.121683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:26.121683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:26.121683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:26.121683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.121683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:26.121683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:26.121683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.121683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:26.121683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:26.121683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:26.121683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.121919 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:26.122157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.122157 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:26.122449 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:26.123203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.123849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:26.123849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.123849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:26.410344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:26.410344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:26.410344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:26.410344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:26.410344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2ce13c62-602b-4159-a60a-7c5b51ca01e4, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:26.410907 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:26.410907 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:26.410907 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:26.410907 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:26.410907 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-ebe02836-594c-4689-86a7-929dc6b874e7, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:26.410907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.411116 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:26.411116 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.411116 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:26.411116 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:26.411116 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.411116 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:26.411116 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.411116 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:26.411538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.411538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:26.411538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:26.411538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:26.411538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.411538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:26.411538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:26.411538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.411538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:26.411538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:26.411538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:26.411538 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.411776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:26.411984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.411984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:26.412391 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:26.412894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.413362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:26.413362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.413362 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:26.417462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:26.417462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:26.417462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:26.417462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:11:26.418711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:26.418949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:26.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:26.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:26.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:26.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.419526 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:26.420339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:26.420887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:26.421346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:26.421678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:26.421678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:26.421678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:26.421678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:26.421678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:26.421678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.421678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:26.421678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:26.421678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.421678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:26.421678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.421678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:26.422343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:26.422874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:26.423457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.423457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:26.423457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:26.423457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.423457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:26.423457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.423457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:26.425730 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:26.426035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:26.426141 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:26.426257 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:26.427247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:26.427247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.427247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:26.427247 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.427247 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:26.427247 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:26.427247 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.427247 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:26.427247 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.427247 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:26.427723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:26.428178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:26.428178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:26.428178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.428178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:26.428178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:26.428178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.428178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:26.428178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.428178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:26.428493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.428493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:26.428493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:26.428493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:26.428493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.428493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:26.428493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:26.428493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:26.428493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:26.428493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.428493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:26.428825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:26.429269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:26.429692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:26.430268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:26.430268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:26.430268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:26.430268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:26.430268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.430268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:26.430268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:26.430268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.430268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:26.430268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.430268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:26.445752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:26.446065 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:26.446177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:26.446384 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:26.446781 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:26.447057 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:26.447379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.447890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:26.447890 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.447890 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:26.447890 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:26.447890 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.447890 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:26.447890 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.447890 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:26.448752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.448752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:26.448752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:26.448752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:26.448752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.448752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:26.448752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:26.448752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.448752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:26.448752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:26.448752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:26.448752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.449033 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:26.449289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.449289 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:26.449466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:26.449767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:26.450136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:26.450136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:26.450136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:31.090849 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:31.091803 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:31.091939 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:31.092621 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:31.131703 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:31.131703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.132260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:31.132260 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.132260 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:31.132260 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:31.132260 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.132260 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:31.132260 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.132260 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:31.133331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.133331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:31.133331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:31.133331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:31.133331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.133331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:31.133331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:31.133331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.133331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:31.133331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:31.133331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:31.133331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.133642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:31.133905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.133905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:31.134394 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:31.134907 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.135423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:31.135423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.135423 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:31.408586 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:31.409419 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:31.409495 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:31.410331 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:31.411600 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:31.411600 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:31.411600 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:31.411600 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:31.411600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.412624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:31.412624 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.412624 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:31.412624 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:31.412624 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.412624 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:31.412624 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.412624 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:31.413352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.413352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:31.413352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:31.413352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:31.413352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.413352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:31.413352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:31.413352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.413352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:31.413352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:31.413352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:31.413352 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.413724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:31.414072 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:31.414503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.415124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:31.415124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:31.415124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.415124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:31.415124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.415124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:31.417319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:31.418921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:31.419043 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:31.419147 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:11:31.420145 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:31.420145 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.420560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:31.420560 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.420560 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:31.420560 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:31.420560 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.420560 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:31.420560 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.420560 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:31.421162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:31.421670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:31.421670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:31.421670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.421670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:31.421670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:31.421670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.421670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:31.421670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.421670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:31.421976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.421976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:31.421976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:31.421976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:31.421976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.421976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:31.421976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:31.421976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:11:31.421976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:31.421976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:31.421976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.421976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:31.422395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:31.422930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.422930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:31.422930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.422930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:31.425204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:31.425544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:31.425878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.425878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:31.425878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:31.425878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.425878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:31.425878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.425878 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:31.426281 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:31.427416 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:31.427508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:31.427624 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:31.427924 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:31.428472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.429200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:31.429200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.429200 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:31.429200 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:31.429200 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.429200 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:31.429200 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.429200 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:31.429793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:31.430094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:31.430094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:31.430094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.430094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:31.430094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:31.430094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.430094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:31.430094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.430094 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:31.430425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.430425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:31.430425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:31.430425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:31.430425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.430425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:31.430425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:31.430425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:31.430425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:31.430425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.430425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:31.430905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.430905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:31.430905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.430905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.431198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:31.431569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:31.431914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.431914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:31.431914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.431914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:31.446170 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:31.446508 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:31.446602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:31.446741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:31.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:31.447393 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:31.447938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.448609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:31.448609 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.448609 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:31.448609 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:31.448609 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.448609 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:31.448609 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.448609 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:31.449652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.449652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:31.449652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:31.449652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:31.449652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.449652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:31.449652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:31.449652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.449652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:31.449652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:31.449652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:31.449652 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.449935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:31.450201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.450201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:31.450444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:31.450771 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:31.451130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:31.451130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:31.451130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:36.091945 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:36.092356 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:36.092356 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:36.093070 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:36.129723 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:36.129890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.130259 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:36.130259 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.130259 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:36.130259 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:36.130259 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.130259 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:36.130259 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.130259 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:36.130855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.130855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:36.130855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:36.130855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:36.130855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.130855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:36.130855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:36.130855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.130855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:36.130855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:36.130855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:36.130855 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.131288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:36.131668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:36.132158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.132723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:36.132723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:36.132723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.132723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:36.132723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.132723 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:36.409460 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:36.409733 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:36.409847 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:36.410675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:36.410821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:36.410959 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:36.411106 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:36.411246 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:36.411312 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:36.411400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.411802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:36.411802 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.411802 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:36.411802 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:36.411802 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.411802 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:36.411802 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.411802 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:36.412441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.412441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:36.412441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:36.412441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:36.412441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.412441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:36.412441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:36.412441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.412441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:36.412441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:36.412441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:36.412441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.412711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:36.412985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.412985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:36.413233 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:36.413978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.414478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:36.414478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.414478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:36.418338 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:36.418698 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:36.418834 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:36.418979 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:11:36.419364 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:36.419534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.419945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:36.419945 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.419945 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:36.419945 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:36.419945 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.419945 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:36.419945 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.419945 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:36.420619 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:36.421016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:36.421016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:36.421016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.421016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:36.421016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:36.421016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.421016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:36.421016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.421016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:36.421290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.421290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:36.421290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:36.421290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:36.421290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.421290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:36.421290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:36.421290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:11:36.421290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:36.421290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:36.421290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.421290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:36.421567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:36.421911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:36.422341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:36.422886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:36.422886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:36.422886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:36.422886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:36.422886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:36.422886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.422886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:36.422886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:36.422886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.422886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:36.422886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.422886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:36.427340 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:36.427529 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:36.427625 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:36.427762 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:36.428041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:36.428280 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.428669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:36.428669 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.428669 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:36.428669 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:36.428669 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.428669 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:36.428669 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.428669 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:36.429361 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:36.429844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:36.429844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:36.429844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.429844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:36.429844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:36.429844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.429844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:36.429844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.429844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:36.430107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.430107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:36.430107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:36.430107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:36.430107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.430107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:36.430107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:36.430107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:36.430107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:36.430107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.430107 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:36.430496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:36.430875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:36.431312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:36.431865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:36.431865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:36.431865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:36.431865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:36.431865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.431865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:36.431865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:36.431865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.431865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:36.431865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.431865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:36.446541 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:36.446937 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:36.447053 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:36.447241 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:36.447505 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:36.447726 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:36.447896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.448344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:36.448344 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.448344 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:36.448344 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:36.448344 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.448344 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:36.448344 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.448344 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:36.448948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.448948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:36.448948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:36.448948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:36.448948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.448948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:36.448948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:36.448948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.448948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:36.448948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:36.448948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:36.448948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.449230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:36.449654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.449654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:36.449699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:36.450052 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:36.450398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:36.450398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:36.450398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:41.092836 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:41.093268 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:41.093560 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:41.094370 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:41.148239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:41.149659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.149659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:41.149659 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.149659 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:41.149659 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:41.149659 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.149659 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:41.149659 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.149659 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:41.150735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.150735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:41.150735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:41.150735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:41.150735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.150735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:41.150735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:41.150735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.150735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:41.150735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:41.150735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:41.150735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.151093 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:41.151357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.151357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:41.152252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:41.152584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.152940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:41.152940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.152940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:41.409877 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:41.410249 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:41.410342 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:41.411067 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:41.411250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:41.411360 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:41.411452 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:41.411561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:41.411655 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:41.411742 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:41.411841 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.412275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:41.412275 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.412275 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:41.412275 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:41.412275 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.412275 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:41.412275 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.412275 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:41.412899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.412899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:41.412899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:41.412899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:41.412899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.412899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:41.412899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:41.412899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.412899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:41.412899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:41.412899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:41.412899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.413140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:41.413650 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:41.414127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.414525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:41.414525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:41.414525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.414525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:41.414525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.414525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:41.418698 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:41.418912 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:41.418989 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:41.419073 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:11:41.419343 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:41.419503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.419838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:41.419838 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.419838 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:41.419838 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:41.419838 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.419838 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:41.419838 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.419838 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:41.420379 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:41.420738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:41.420738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:41.420738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.420738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:41.420738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:41.420738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.420738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:41.420738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.420738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:41.420924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.420924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:41.420924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:41.420924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:41.420924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.420924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:41.420924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:41.420924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:11:41.420924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:41.420924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:41.420924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.420924 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:41.421177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:41.421440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.421440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:41.421440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.421440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:41.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:41.421926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:41.422268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.422268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:41.422268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:41.422268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.422268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:41.422268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.422268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:41.427671 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:41.427870 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:41.427976 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:41.428068 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:41.428284 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:41.428429 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.428713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:41.428713 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.428713 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:41.428713 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:41.428713 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.428713 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:41.428713 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.428713 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:41.429171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:41.429467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:41.429467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:41.429467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.429467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:41.429467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:41.429467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.429467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:41.429467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.429467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:41.429725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.429725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:41.429725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:41.429725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:41.429725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.429725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:41.429725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:41.429725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:41.429725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:41.429725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.429725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:41.429951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:41.430205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.430205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:41.430205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.430205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.430545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:41.430916 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:41.431270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.431270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:41.431270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.431270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:41.446911 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:41.447251 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:41.447358 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:41.447557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:41.447811 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:41.447977 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:41.448219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.448610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:41.448610 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.448610 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:41.448610 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:41.448610 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.448610 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:41.448610 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.448610 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:41.449541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.449541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:41.449541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:41.449541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:41.449541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.449541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:41.449541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:41.449541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.449541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:41.449541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:41.449541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:41.449541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.449797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:41.450117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.450117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:41.450692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:41.451075 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:41.451483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:41.451483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:41.451483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:46.093247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:46.093652 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:46.093652 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:46.094140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:46.137868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:46.138041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:46.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:46.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:46.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:46.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.138574 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:46.139705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.139705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:46.139705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:46.139705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:46.139705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.139705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:46.139705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:46.139705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.139705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:46.139705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:46.139705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:46.139705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.140278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:46.140583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.140583 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:46.141076 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:46.141615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.142117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:46.142117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.142117 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:46.410330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:46.410783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:46.410943 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:46.411773 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:46.411970 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:46.412132 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:46.412254 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:46.412375 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:46.412474 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:46.412623 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:46.412786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.413435 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:46.413435 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.413435 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:46.413435 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:46.413435 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.413435 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:46.413435 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.413435 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:46.414169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.414169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:46.414169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:46.414169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:46.414169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.414169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:46.414169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:46.414169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.414169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:46.414169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:46.414169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:46.414169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.414739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:46.415097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:46.415549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.416178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:46.416178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:46.416178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.416178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:46.416178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.416178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:46.419058 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:46.419572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:46.419572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:46.419572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:11:46.419977 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:46.420640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.420862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:46.420862 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.420862 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:46.420862 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:46.420862 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.420862 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:46.420862 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.420862 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:46.421407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:46.421825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:46.421825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:46.421825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.421825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:46.421825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:46.421825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.421825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:46.421825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.421825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:46.422110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.422110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:46.422110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:46.422110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:46.422110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.422110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:46.422110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:46.422110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:11:46.422110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:46.422110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:46.422110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.422110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:46.422682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:46.423332 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:46.423746 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:46.424274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:46.424274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:46.424274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:46.424274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:46.424274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:46.424274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.424274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:46.424274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:46.424274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.424274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:46.424274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.424274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:46.428039 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:46.428354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:46.428751 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:46.429074 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:46.429164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.430442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:46.430986 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:46.431475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.431819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:46.432201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:46.432608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:46.433136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:46.433136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.433136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:46.433136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:46.433136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.433136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:46.433136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.433136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:46.447344 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:46.447805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:46.447908 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:46.448058 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:46.449484 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:46.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:46.451310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:46.451702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:46.451702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:46.451702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.451702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:46.451702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:46.451702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.451702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:46.451702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.451702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:46.451702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.451702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:46.451702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:46.452049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:46.452636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:51.095209 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:51.095209 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:51.095209 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:51.095209 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:51.126542 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:51.126727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.127268 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:51.127268 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.127268 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:51.127268 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:51.127268 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.127268 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:51.127268 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.127268 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:51.127952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.127952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:51.127952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:51.127952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:51.127952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.127952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:51.127952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:51.127952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.127952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:51.127952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:51.127952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:51.127952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.128371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:51.128622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.128622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:51.128821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:51.129624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.130589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:51.130589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.130589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:51.410828 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:51.411035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:51.411110 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:51.411941 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:51.412074 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:51.412233 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:51.412354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:51.412493 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:51.412627 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:51.412759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:51.412891 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.413408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:51.413408 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.413408 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:51.413408 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:51.413408 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.413408 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:51.413408 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.413408 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:51.414005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.414005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:51.414005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:51.414005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:51.414005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.414005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:51.414005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:51.414005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.414005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:51.414005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:51.414005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:51.414005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.414316 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:51.414541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.414541 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:51.414839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:51.415173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.415568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:51.415568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.415568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:51.419391 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:51.419538 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:51.419615 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:51.419708 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:11:51.419974 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:51.420146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.420564 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:51.420564 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.420564 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:51.420564 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:51.420564 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.420564 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:51.420564 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.420564 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:51.421127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:51.421450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:51.421450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:51.421450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.421450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:51.421450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:51.421450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.421450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:51.421450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.421450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:51.421725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.421725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:51.421725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:51.421725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:51.421725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.421725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:51.421725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:51.421725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:11:51.421725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:51.421725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:51.421725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.421725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:51.422230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:51.422744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:51.423262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:51.423578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:51.423578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:51.423578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:51.423578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:51.423578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:51.423578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.423578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:51.423578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:51.423578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.423578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:51.423578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.423578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:51.428377 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:51.428539 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:51.428618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:51.428710 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:51.428960 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:51.429227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.429571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:51.429571 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.429571 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:51.429571 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:51.429571 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.429571 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:51.429571 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.429571 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:51.430054 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:51.430384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:51.430384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:51.430384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.430384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:51.430384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:51.430384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.430384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:51.430384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.430384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:51.430720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.430720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:51.430720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:51.430720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:51.430720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.430720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:51.430720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:51.430720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:51.430720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:51.430720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.430720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:51.430920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:51.431227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.431227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:51.431227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.431227 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.431498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:51.431790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:51.432136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.432136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:51.432136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.432136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:51.447868 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:51.448140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:51.448277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:51.448482 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:51.448889 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:51.449133 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:51.449378 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.449992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:51.449992 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.449992 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:51.449992 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:51.449992 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.449992 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:51.449992 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.449992 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:51.450669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.450669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:51.450669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:51.450669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:51.450669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.450669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:51.450669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:51.450669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.450669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:51.450669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:51.450669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:51.450669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.451071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:51.451462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.451462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:51.451819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:51.452354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:51.452860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:51.452860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:51.452860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:56.094476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:56.094476 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:56.094862 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:56.095897 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:56.136909 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:56.137066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.137797 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:56.137797 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.137797 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:56.137797 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:56.137797 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.137797 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:56.137797 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.137797 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:56.138545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.138545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:56.138545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:56.138545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:56.138545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.138545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:56.138545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:56.138545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.138545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:56.138545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:56.138545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:56.138545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.138808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:56.139015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.139015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:56.139463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:56.139851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.141442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:56.141442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.141442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:56.411224 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:56.411426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:56.411501 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:56.412156 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:56.412301 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2168c57b-e149-43e1-b184-bf21d9c205cd, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:56.412405 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:56.412496 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:56.412579 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:56.412669 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8b5061d7-f426-4fdf-8094-997a01b6252a, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:56.412754 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:56.412850 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.413256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:56.413256 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.413256 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:56.413256 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:56.413256 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.413256 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:56.413256 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.413256 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:56.413813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.413813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:56.413813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:56.413813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:56.413813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.413813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:56.413813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:56.413813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.413813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:56.413813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:56.413813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:56.413813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.414420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:56.414976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:56.415598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.415888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:56.415888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:56.415888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.415888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:56.415888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.415888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:56.419610 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:56.419741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:56.419813 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:56.419895 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:11:56.420121 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:56.420269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.420569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:56.420569 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.420569 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:56.420569 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:56.420569 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.420569 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:56.420569 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.420569 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:56.421017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:56.421286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:56.421286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:56.421286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.421286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:56.421286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:56.421286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.421286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:56.421286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.421286 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:56.421615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.421615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:56.421615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:56.421615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:56.421615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.421615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:56.421615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:56.421615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:11:56.421615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:56.421615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:56.421615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.421615 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:56.422217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:56.422570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:56.422780 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:56.423074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:56.423074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:56.423074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:56.423074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:56.423074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:56.423074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.423074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:56.423074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:56.423074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.423074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:56.423074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.423074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:56.428588 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:56.428744 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:56.428819 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:56.428920 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:56.429134 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:56.429302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.429605 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:56.429605 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.429605 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:56.429605 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:56.429605 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.429605 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:56.429605 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.429605 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:11:56.430174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:11:56.430443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:56.430443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:11:56.430443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.430443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:11:56.430443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:11:56.430443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.430443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:11:56.430443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.430443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:11:56.430783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.430783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:56.430783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:56.430783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:56.430783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.430783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:56.430783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:56.430783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:56.430783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:56.430783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.430783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:56.431393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:56.431937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:56.432202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:56.432486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:56.432486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:56.432486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:56.432486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:56.432486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.432486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:56.432486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:56.432486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.432486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:56.432486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.432486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:11:56.448162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:11:56.448335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:11:56.448411 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:11:56.448533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:11:56.448762 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:56.448923 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:11:56.449098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.449451 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:11:56.449451 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.449451 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:11:56.449451 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:11:56.449451 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.449451 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:11:56.449451 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.449451 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:11:56.450025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.450025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:11:56.450025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:56.450025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:11:56.450025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.450025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:56.450025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:56.450025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.450025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:56.450025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:56.450025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:11:56.450025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.450274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:11:56.450486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.450486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:11:56.450730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:11:56.451508 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:11:56.451972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:11:56.451972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:11:56.451972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:01.094596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:01.095550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:01.095550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:01.095550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:01.127794 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:01.127794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.127949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:01.127949 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.127949 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:01.127949 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:01.127949 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.127949 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:01.127949 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.127949 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:01.128585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.128585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:01.128585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:01.128585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:01.128585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.128585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:01.128585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:01.128585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.128585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:01.128585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:01.128585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:01.128585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.129389 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:01.129894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:01.130202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.130633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:01.130633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:01.130633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.130633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:01.130633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.130633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:01.411569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:01.412763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:01.412763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:01.413217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:01.413523 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:01.413795 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:01.414078 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:01.414935 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:01.414935 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:01.416207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:01.416674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.417017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:01.417017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:01.417017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.417017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:01.417017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.417017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:01.418516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:01.419651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.422084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:01.422084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.422084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:01.422084 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:01.422084 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:01.422084 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:01.422084 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:12:01.422084 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:01.422465 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.422787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:01.422787 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.422787 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:01.422787 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:01.422787 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.422787 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:01.422787 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.422787 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:01.425370 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:01.425831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:01.426262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:01.426600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:01.426600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:01.426600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:01.426600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:01.426600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:01.426600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.426600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:01.426600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:01.426600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.426600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:01.426600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.426600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:01.428718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:01.430825 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:01.432069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.432069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:01.432069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:01.432069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.432069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:01.432069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.432069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:01.432069 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:01.432069 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:01.432069 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:01.432495 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:01.432495 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:01.432495 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.435211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.436078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:01.438603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.438603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:01.438603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:01.438603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:01.438603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.438603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:01.438603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:01.438603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:01.438603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:01.438603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.438603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:01.439064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:01.439553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:01.440042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:01.440683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:01.440683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:01.440683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:01.440683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:01.440683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.440683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:01.440683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:01.440683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.440683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:01.440683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.440683 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:01.448613 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:01.449303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:01.449303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:01.449303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:01.449623 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:01.449883 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:01.450136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.450579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:01.450579 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.450579 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:01.450579 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:01.450579 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.450579 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:01.450579 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.450579 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:01.452572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.452572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:01.452572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:01.452572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:01.452572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.452572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:01.452572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:01.452572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.452572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:01.452572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:01.452572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:01.452572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.452772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:01.453000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:01.453308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:01.453624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:01.453624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:01.453624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:01.453624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:01.453624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:01.453624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:06.094972 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:06.095430 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:06.095841 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:06.097394 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:06.140539 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:06.140539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.140879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:06.140879 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.140879 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:06.140879 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:06.140879 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.140879 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:06.140879 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.140879 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:06.141805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.141805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:06.141805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:06.141805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:06.141805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.141805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:06.141805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:06.141805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.141805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:06.141805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:06.141805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:06.141805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.142064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:06.142315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.142315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:06.142763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:06.143053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.143397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:06.143397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.143397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:06.412205 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:06.413426 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:06.413535 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:06.415303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:06.415303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:06.415303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:06.415303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:06.415303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:06.415303 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.415477 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:06.415477 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.415477 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:06.415477 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:06.415477 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.415477 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:06.415477 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.415477 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:06.416056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.416056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:06.416056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:06.416056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:06.416056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.416056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:06.416056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:06.416056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.416056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:06.416056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:06.416056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:06.416056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.416344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:06.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.416612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:06.416774 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:06.417067 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.417410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:06.417410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.417410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:06.420238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:06.420395 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:06.420469 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:06.420557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:12:06.421160 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:06.421160 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.421266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:06.421266 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.421266 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:06.421266 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:06.421266 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.421266 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:06.421266 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.421266 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:06.421761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:06.422078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:06.422078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:06.422078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.422078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:06.422078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:06.422078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.422078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:06.422078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.422078 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:06.422308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.422308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:06.422308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:06.422308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:06.422308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.422308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:06.422308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:06.422308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:12:06.422308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:06.422308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:06.422308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.422308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:06.422553 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:06.422808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.422808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:06.422808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.422808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:06.422952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:06.423278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:06.423610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.423610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:06.423610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:06.423610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.423610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:06.423610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.423610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:06.429250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:06.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:06.429628 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:06.429719 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:06.430401 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:06.430401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.430503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:06.430503 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.430503 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:06.430503 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:06.430503 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.430503 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:06.430503 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.430503 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:06.430992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:06.431302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:06.431302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:06.431302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.431302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:06.431302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:06.431302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.431302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:06.431302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.431302 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:06.431620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.431620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:06.431620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:06.431620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:06.431620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.431620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:06.431620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:06.431620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:06.431620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:06.431620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.431620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:06.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:06.432136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.432136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:06.432136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.432136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.432324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:06.432631 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:06.434350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.434350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:06.434350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.434350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:06.449036 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:06.449246 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:06.449326 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:06.449463 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:06.450945 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:06.450945 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:06.450945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.451488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:06.451488 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.451488 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:06.451488 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:06.451488 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.451488 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:06.451488 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.451488 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:06.452220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.452220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:06.452220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:06.452220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:06.452220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.452220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:06.452220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:06.452220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.452220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:06.452220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:06.452220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:06.452220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.452685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:06.453139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:06.453711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:06.454381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:06.454381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:06.454381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:06.454381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:06.454381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:06.454381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:11.095359 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:11.095883 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:11.095883 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:11.096673 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:11.128801 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:11.129022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.129580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:11.129580 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.129580 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:11.129580 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:11.129580 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.129580 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:11.129580 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.129580 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:11.130284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.130284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:11.130284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:11.130284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:11.130284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.130284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:11.130284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:11.130284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.130284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:11.130284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:11.130284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:11.130284 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.130680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:11.130993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.130993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:11.131065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:11.131452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:11.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.131902 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:11.413609 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:11.413823 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:11.413881 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:11.414991 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:11.415202 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:11.415348 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:11.415461 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:11.415743 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:11.415743 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:11.417632 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:11.418010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:11.418726 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:11.419178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:11.419178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:11.419178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:11.419178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.419178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:11.419178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:11.419178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.419178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:11.419178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.419178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:11.420757 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:11.421029 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:11.421124 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:11.421227 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:12:11.421852 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:11.422011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.422467 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:11.422467 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.422467 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:11.422467 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:11.422467 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.422467 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:11.422467 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.422467 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:11.422989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:11.423612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:11.424137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:11.424676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:11.424676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:11.424676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:11.424676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:11.424676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:11.424676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.424676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:11.424676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:11.424676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.424676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:11.424676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.424676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:11.425210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:11.425592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:11.426178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.426178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:11.426178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:11.426178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.426178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:11.426178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.426178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:11.429688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:11.429913 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:11.430181 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:11.430181 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:11.431164 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:11.431400 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.432381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:11.432381 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.432381 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:11.432381 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:11.432381 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.432381 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:11.432381 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.432381 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:11.433041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:11.433342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:11.433342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:11.433342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.433342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:11.433342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:11.433342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.433342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:11.433342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.433342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:11.433747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.433747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:11.433747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:11.433747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:11.433747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.433747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:11.433747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:11.433747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:11.433747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:11.433747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.433747 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:11.434343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:11.434714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:11.435432 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:11.436549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:11.436549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:11.436549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:11.436549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:11.436549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.436549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:11.436549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:11.436549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.436549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:11.436549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.436549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:11.449372 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:11.449724 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:11.449783 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:11.449944 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:11.451367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:11.451553 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:11.451703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.452182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:11.452182 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.452182 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:11.452182 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:11.452182 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.452182 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:11.452182 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.452182 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:11.452982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.452982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:11.452982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:11.452982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:11.452982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.452982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:11.452982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:11.452982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.452982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:11.452982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:11.452982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:11.452982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.454079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:11.454557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:11.455091 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:11.455790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:11.455790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:11.455790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:11.455790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:11.455790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:11.455790 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:16.095804 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:16.096363 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:16.096480 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:16.097297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:16.133691 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:16.133911 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:16.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:16.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:16.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:16.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.134439 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:16.135092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.135092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:16.135092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:16.135092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:16.135092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.135092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:16.135092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:16.135092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.135092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:16.135092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:16.135092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:16.135092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.135348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:16.135559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.135559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:16.135860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:16.136598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.136879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:16.136879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.136879 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:16.413962 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:16.414400 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:16.414552 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:16.415225 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:16.415395 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:16.415514 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:16.415649 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:16.415758 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:16.415856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.416310 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:16.416310 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.416310 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:16.416310 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:16.416310 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.416310 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:16.416310 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.416310 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:16.416982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.416982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:16.416982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:16.416982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:16.416982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.416982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:16.416982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:16.416982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.416982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:16.416982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:16.416982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:16.416982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.417274 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:16.417535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.417535 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:16.417789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:16.418561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.419071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:16.419071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.419071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:16.421070 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:16.421316 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:16.421430 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:16.421548 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:12:16.421856 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:16.422005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.422437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:16.422437 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.422437 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:16.422437 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:16.422437 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.422437 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:16.422437 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.422437 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:16.423272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:16.423511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:16.423511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:16.423511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.423511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:16.423511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:16.423511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.423511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:16.423511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.423511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:16.424042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.424042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:16.424042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:16.424042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:16.424042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.424042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:16.424042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:16.424042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:12:16.424042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:16.424042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:16.424042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.424042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:16.424294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:16.424534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.424534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:16.424534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.424534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:16.425308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:16.425718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:16.426172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.426172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:16.426172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:16.426172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.426172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:16.426172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.426172 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:16.433202 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:16.433202 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:16.433202 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:16.433202 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:16.433202 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:16.433202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.433920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:16.434412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:16.434829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.435169 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:16.435521 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:16.435869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:16.437058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:16.437058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.437058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:16.437058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:16.437058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.437058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:16.437058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.437058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:16.449770 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:16.450041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:16.450041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:16.450304 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:16.450590 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:16.450787 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:16.450978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:16.452831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:16.453978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.454625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:16.454955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:16.455494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:16.455494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:16.455494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:16.455494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:16.455494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:16.455494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:16.455494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:16.455494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:16.455494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:16.455494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:21.096289 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:21.096711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:21.096711 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:21.097234 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:21.111877 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=1076,neutron:UPDATE=2,neutron:DELETE=12 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:12:21.126159 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:21.126408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.126909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:21.126909 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.126909 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:21.126909 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:21.126909 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.126909 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:21.126909 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.126909 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:21.127576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.127576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:21.127576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:21.127576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:21.127576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.127576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:21.127576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:21.127576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.127576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:21.127576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:21.127576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:21.127576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.127875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:21.128273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:21.128584 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.128959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:21.128959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:21.128959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.128959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:21.128959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.128959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:21.414583 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:21.415581 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:21.415581 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:21.415858 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:21.416063 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:21.416305 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:21.416425 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:21.416565 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:21.416670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:21.418248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:21.418685 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.418947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:21.418947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:21.418947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.418947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:21.418947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.418947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:21.419860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:21.420209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:21.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.420753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:21.421581 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:21.422172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:21.422172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:21.422172 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:12:21.422462 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:21.422623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.423031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:21.423031 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.423031 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:21.423031 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:21.423031 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.423031 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:21.423031 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.423031 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:21.423889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:21.424198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:21.424198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:21.424198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.424198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:21.424198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:21.424198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.424198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:21.424198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.424198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:21.424954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.424954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:21.424954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:21.424954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:21.424954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.424954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:21.424954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:21.424954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:12:21.424954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:21.424954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:21.424954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.424954 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:21.425253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:21.425480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.425480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:21.425480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.425480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:21.426386 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:21.426706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:21.427058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.427058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:21.427058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:21.427058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.427058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:21.427058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.427058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:21.430335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:21.430821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:21.430821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:21.430821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:21.431013 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:21.431200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.433469 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:21.433853 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:21.434149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.434420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:21.434630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:21.434939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:21.435359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:21.435359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.435359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:21.435359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:21.435359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.435359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:21.435359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.435359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:21.450205 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:21.450777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:21.450777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:21.450777 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:21.451173 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:21.451385 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:21.451547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:21.453334 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:21.453699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.453972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:21.454265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:21.454610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:21.454610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:21.454610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:21.454610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:21.454610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:21.454610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:21.454610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:21.454610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:21.454610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:21.454610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:26.096775 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:26.097499 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:26.097499 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:26.097741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:26.124593 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:26.124753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.125306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:26.125306 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.125306 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:26.125306 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:26.125306 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.125306 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:26.125306 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.125306 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:26.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:26.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:26.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:26.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:26.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:26.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:26.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:26.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:26.125950 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.126369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:26.126667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.126667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:26.126725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:26.127089 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.127556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:26.127556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.127556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:26.415030 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:26.415540 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:26.415700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:26.416570 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:26.416749 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:26.416898 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:26.417047 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:26.417217 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:26.417365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.417849 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:26.417849 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.417849 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:26.417849 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:26.417849 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.417849 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:26.417849 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.417849 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:26.418603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.418603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:26.418603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:26.418603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:26.418603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.418603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:26.418603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:26.418603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.418603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:26.418603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:26.418603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:26.418603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.418938 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:26.419327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.419327 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:26.419725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:26.420011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.420346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:26.420346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.420346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:26.421986 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:26.422250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:26.422387 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:26.422480 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:12:26.422723 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:26.422898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.423354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:26.423354 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.423354 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:26.423354 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:26.423354 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.423354 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:26.423354 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.423354 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:26.424322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:26.424735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:26.424735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:26.424735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.424735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:26.424735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:26.424735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.424735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:26.424735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.424735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:26.424984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.424984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:26.424984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:26.424984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:26.424984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.424984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:26.424984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:26.424984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:12:26.424984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:26.424984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:26.424984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.424984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:26.425337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:26.425712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.425712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:26.425712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.425712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:26.426570 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:26.427404 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:26.428017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.428017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:26.428017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:26.428017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.428017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:26.428017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.428017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:26.430654 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:26.430911 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:26.431041 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:26.431244 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:26.431601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:26.431741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.432143 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:26.432143 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.432143 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:26.432143 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:26.432143 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.432143 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:26.432143 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.432143 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:26.432882 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:26.433292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:26.433292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:26.433292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.433292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:26.433292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:26.433292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.433292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:26.433292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.433292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:26.433534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.433534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:26.433534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:26.433534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:26.433534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.433534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:26.433534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:26.433534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:26.433534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:26.433534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.433534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:26.433818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:26.434173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.434173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:26.434173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.434173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.435015 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:26.436048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:26.437276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.437276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:26.437276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.437276 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:26.450606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:26.450960 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:26.451078 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:26.451302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:26.451596 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:26.451825 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:26.452023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.452714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:26.452714 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.452714 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:26.452714 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:26.452714 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.452714 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:26.452714 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.452714 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:26.453572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.453572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:26.453572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:26.453572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:26.453572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.453572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:26.453572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:26.453572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.453572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:26.453572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:26.453572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:26.453572 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.453953 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:26.454382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.454382 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:26.454763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:26.455684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:26.456752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:26.456752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:26.456752 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:31.097170 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:31.099603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:31.099603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:31.099603 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:31.133182 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:31.133576 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.134168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:31.134168 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.134168 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:31.134168 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:31.134168 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.134168 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:31.134168 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.134168 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:31.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:31.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:31.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:31.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:31.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:31.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:31.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:31.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:31.135342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.135753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:31.136124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.136124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:31.137341 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:31.137852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.138824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:31.138824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.138824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:31.416171 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:31.417336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:31.417336 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:31.417750 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:31.418014 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:31.418453 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:31.418733 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:31.418979 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:31.420914 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.421348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:31.421348 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.421348 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:31.421348 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:31.421348 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.421348 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:31.421348 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.421348 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:31.422662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.422662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:31.422662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:31.422662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:31.422662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.422662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:31.422662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:31.422662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.422662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:31.422662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:31.422662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:31.422662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.422965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:31.423198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.423198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:31.423739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:31.424520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.424840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:31.424840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.424840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:31.424840 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:31.424840 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:31.424840 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:31.424840 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:12:31.425561 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:31.425561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.425731 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:31.425731 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.425731 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:31.425731 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:31.425731 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.425731 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:31.425731 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.425731 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:31.426390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:31.426693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:31.426693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:31.426693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.426693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:31.426693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:31.426693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.426693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:31.426693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.426693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:31.430063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.430063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:31.430063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:31.430063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:31.430063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.430063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:31.430063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:31.430063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:12:31.430063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:31.430063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:31.430063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.430063 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:31.430418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.430651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:31.431124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:31.431453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:31.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:31.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:31.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:31.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.431860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:31.432101 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:31.433759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:31.433759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:31.433759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:31.434225 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:31.438213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.438865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:31.439175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.439175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:31.439175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:31.439175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:31.439175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.439175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:31.439175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:31.439175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:31.439175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:31.439175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.439175 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:31.439416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:31.439669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.439669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:31.439669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.439669 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.439786 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:31.440343 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:31.440701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.440701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:31.440701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.440701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:31.450956 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:31.451634 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:31.451634 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:31.451634 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:31.451921 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:31.452201 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:31.452448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.453006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:31.453006 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.453006 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:31.453006 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:31.453006 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.453006 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:31.453006 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.453006 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:31.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:31.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:31.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:31.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:31.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:31.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:31.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:31.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:31.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.454105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:31.454398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.454398 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:31.455228 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:31.455582 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:31.455901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:31.455901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:31.455901 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:36.098035 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:36.099957 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:36.099957 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:36.099957 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:36.124972 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:36.125313 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.125763 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:36.125763 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.125763 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:36.125763 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:36.125763 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.125763 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:36.125763 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.125763 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:36.126545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.126545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:36.126545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:36.126545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:36.126545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.126545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:36.126545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:36.126545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.126545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:36.126545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:36.126545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:36.126545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.126804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:36.127014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.127014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:36.127887 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:36.128414 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.129040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:36.129040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.129040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:36.416780 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:36.417808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:36.417808 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:36.418156 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:36.418403 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:36.418632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:36.418828 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:36.419021 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:36.419251 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.419719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:36.419719 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.419719 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:36.419719 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:36.419719 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.419719 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:36.419719 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.419719 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:36.421453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.421453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:36.421453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:36.421453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:36.421453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.421453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:36.421453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:36.421453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.421453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:36.421453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:36.421453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:36.421453 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.421799 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:36.422162 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:36.422452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.422782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:36.422782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:36.422782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.422782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:36.422782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.422782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:36.424159 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:36.424621 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:36.424621 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:36.424621 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:12:36.424916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:36.425159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.427872 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:36.428317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:36.428581 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.428874 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:36.429226 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:36.429590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:36.430040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:36.430040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:36.430040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.430040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:36.430040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:36.430040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.430040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:36.430040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.430040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:36.432428 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:36.432664 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:36.432818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:36.432978 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:36.433332 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:36.433624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.434128 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:36.434128 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.434128 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:36.434128 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:36.434128 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.434128 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:36.434128 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.434128 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:36.434877 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:36.435164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:36.435164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:36.435164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.435164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:36.435164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:36.435164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.435164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:36.435164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.435164 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:36.435952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.435952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:36.435952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:36.435952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:36.435952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.435952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:36.435952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:36.435952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:36.435952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:36.435952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.435952 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:36.436202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:36.436481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.436481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:36.436481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.436481 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.437320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:36.437712 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:36.438209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.438209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:36.438209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.438209 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:36.451319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:36.452016 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:36.452016 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:36.452016 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:36.452391 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:36.452651 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:36.452894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:36.454675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:36.455021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.455307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:36.455544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:36.455894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:36.455894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:36.455894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:36.455894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:36.455894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:36.455894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:36.455894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:36.455894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:36.455894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:36.455894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:41.098915 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:41.099412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:41.099671 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:41.101319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:41.145688 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:41.145936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.146480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:41.146480 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.146480 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:41.146480 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:41.146480 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.146480 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:41.146480 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.146480 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:41.147741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.147741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:41.147741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:41.147741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:41.147741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.147741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:41.147741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:41.147741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.147741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:41.147741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:41.147741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:41.147741 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.148098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:41.148397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.148397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:41.148839 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:41.149294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.149695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:41.149695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.149695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:41.417390 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:41.417821 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:41.417916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:41.419224 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:41.419473 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:41.419908 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:41.420052 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:41.420223 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:41.420624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.421642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:41.421642 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.421642 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:41.421642 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:41.421642 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.421642 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:41.421642 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.421642 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:41.422934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.422934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:41.422934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:41.422934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:41.422934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.422934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:41.422934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:41.422934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.422934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:41.422934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:41.422934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:41.422934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.423270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:41.423475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.423475 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:41.424295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:41.424580 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.424884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:41.424884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.424884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:41.425090 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:41.425700 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:41.425836 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:41.425940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:12:41.426204 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:41.426707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.427373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:41.427373 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.427373 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:41.427373 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:41.427373 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.427373 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:41.427373 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.427373 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:41.427941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:41.428243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:41.428243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:41.428243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.428243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:41.428243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:41.428243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.428243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:41.428243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.428243 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:41.429503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.429503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:41.429503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:41.429503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:41.429503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.429503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:41.429503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:41.429503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:12:41.429503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:41.429503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:41.429503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.429503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:41.429756 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:41.429978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.429978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:41.429978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.429978 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:41.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:41.431027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:41.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:41.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:41.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:41.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:41.432752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:41.433097 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:41.433246 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:41.433632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:41.433895 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:41.434377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:41.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:41.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:41.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:41.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:41.435544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:41.435796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:41.435796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:41.435796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.435796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:41.435796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:41.435796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.435796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:41.435796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.435796 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:41.436082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.436082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:41.436082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:41.436082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:41.436082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.436082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:41.436082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:41.436082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:41.436082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:41.436082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.436082 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:41.436338 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:41.436559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.436559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:41.436559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.436559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.438290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:41.438607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:41.438922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.438922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:41.438922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.438922 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:41.452569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:41.452569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:41.452569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:41.452569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:41.452569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:41.452865 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:41.453057 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.456623 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:41.456623 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.456623 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:41.456623 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:41.456623 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.456623 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:41.456623 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.456623 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:41.460211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.460211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:41.460211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:41.460211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:41.460211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.460211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:41.460211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:41.460211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.460211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:41.460211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:41.460211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:41.460211 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.460654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:41.461005 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:41.461462 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:41.461964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:41.461964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:41.461964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:41.461964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:41.461964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:41.461964 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:46.099407 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:46.099789 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:46.099789 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:46.100578 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:46.128053 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:46.128373 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:46.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:46.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:46.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:46.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.128795 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:46.129577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.129577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:46.129577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:46.129577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:46.129577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.129577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:46.129577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:46.129577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.129577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:46.129577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:46.129577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:46.129577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.129834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:46.130061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.130061 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:46.130805 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:46.131104 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.131472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:46.131472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.131472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:46.417938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:46.418109 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:46.418252 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:46.418953 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:46.419153 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:46.419335 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:46.419490 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:46.419642 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:46.419823 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.420290 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:46.420290 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.420290 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:46.420290 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:46.420290 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.420290 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:46.420290 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.420290 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:46.421011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.421011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:46.421011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:46.421011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:46.421011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.421011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:46.421011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:46.421011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.421011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:46.421011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:46.421011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:46.421011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.421282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:46.421497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.421497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:46.422315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:46.422620 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.422960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:46.422960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.422960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:46.425838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:46.425901 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:46.426001 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:46.426087 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:12:46.426388 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:46.426579 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.426951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:46.426951 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.426951 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:46.426951 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:46.426951 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.426951 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:46.426951 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.426951 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:46.427610 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:46.427871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:46.427871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:46.427871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.427871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:46.427871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:46.427871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.427871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:46.427871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.427871 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:12:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.428478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:46.428681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:46.428889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.428889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:46.428889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.428889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:46.429127 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:46.429393 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:46.429670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.429670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:46.429670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:46.429670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.429670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:46.429670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.429670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:46.433131 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:46.433323 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:46.433490 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:46.434200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:46.434200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:46.434200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.434498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:46.434498 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.434498 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:46.434498 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:46.434498 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.434498 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:46.434498 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.434498 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:46.435079 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:46.435337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:46.435337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:46.435337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.435337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:46.435337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:46.435337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.435337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:46.435337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.435337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:46.435642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.435642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:46.435642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:46.435642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:46.435642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.435642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:46.435642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:46.435642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:46.435642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:46.435642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.435642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:46.435818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:46.436020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.436020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:46.436020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.436020 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.438249 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:46.438644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:46.439004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.439004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:46.439004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.439004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:46.451954 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:46.452177 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:46.452365 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:46.452556 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:46.452819 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:46.453236 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:46.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:46.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:46.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:46.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:46.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.453851 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:46.454479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.454479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:46.454479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:46.454479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:46.454479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.454479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:46.454479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:46.454479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.454479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:46.454479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:46.454479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:46.454479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.454720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:46.454923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.454923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:46.455202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:46.455460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:46.455794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:46.455794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:46.455794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:51.099870 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:51.100534 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:51.100534 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:51.100996 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:51.127940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:51.128235 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.128735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:51.128735 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.128735 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:51.128735 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:51.128735 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.128735 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:51.128735 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.128735 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:51.129703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.129703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:51.129703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:51.129703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:51.129703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.129703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:51.129703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:51.129703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.129703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:51.129703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:51.129703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:51.129703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.130183 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:51.130762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.130762 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:51.131534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:51.132008 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.132395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:51.132395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.132395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:51.418293 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:51.418558 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:51.418649 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:51.419387 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:51.419584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:51.419729 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:51.419865 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:51.420002 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:51.420140 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.420642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:51.420642 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.420642 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:51.420642 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:51.420642 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.420642 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:51.420642 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.420642 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:51.421368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.421368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:51.421368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:51.421368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:51.421368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.421368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:51.421368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:51.421368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.421368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:51.421368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:51.421368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:51.421368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.421644 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:51.421851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.421851 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:51.422574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:51.422866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.423230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:51.423230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.423230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:51.425930 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:51.426113 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:51.426231 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:51.426324 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:12:51.426645 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:51.426827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.427299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:51.427299 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.427299 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:51.427299 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:51.427299 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.427299 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:51.427299 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.427299 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:51.427869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:51.428173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:51.428173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:51.428173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.428173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:51.428173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:51.428173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.428173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:51.428173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.428173 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:51.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:51.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:51.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:51.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:51.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:51.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:12:51.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:51.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:51.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:51.429011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:51.429287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.429287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:51.429287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.429287 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:51.429870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:51.430216 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:51.430552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.430552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:51.430552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:51.430552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.430552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:51.430552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.430552 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:51.433389 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:51.433560 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:51.433647 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:51.433739 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:51.433992 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:51.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.434734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:51.434734 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.434734 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:51.434734 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:51.434734 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.434734 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:51.434734 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.434734 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:51.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:51.435690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:51.435690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:51.435690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.435690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:51.435690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:51.435690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.435690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:51.435690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.435690 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:51.436354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.436354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:51.436354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:51.436354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:51.436354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.436354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:51.436354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:51.436354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:51.436354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:51.436354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.436354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:51.436574 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:51.436816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.436816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:51.436816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.436816 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.437525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:51.437951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:51.438345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.438345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:51.438345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.438345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:51.452274 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:51.452584 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:51.452713 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:51.452877 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:51.453238 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:51.453522 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:51.453748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.454181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:51.454181 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.454181 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:51.454181 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:51.454181 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.454181 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:51.454181 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.454181 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:51.454993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.454993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:51.454993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:51.454993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:51.454993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.454993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:51.454993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:51.454993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.454993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:51.454993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:51.454993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:51.454993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.455474 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:51.455883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.455883 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:51.456550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:51.457146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:51.457781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:51.457781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:51.457781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:56.100178 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:56.100752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:56.100752 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:56.101230 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:56.133239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:56.133466 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.134017 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:56.134017 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.134017 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:56.134017 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:56.134017 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.134017 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:56.134017 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.134017 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:56.137208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.137208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:56.137208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:56.137208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:56.137208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.137208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:56.137208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:56.137208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.137208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:56.137208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:56.137208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:56.137208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.137647 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:56.138029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:56.138555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.139139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:56.139139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:56.139139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.139139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:56.139139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.139139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:56.418649 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:56.419080 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:56.419286 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:56.420162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:56.420373 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:56.420531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:56.420682 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:56.420838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:56.420968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.421548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:56.421548 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.421548 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:56.421548 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:56.421548 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.421548 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:56.421548 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.421548 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:56.422504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.422504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:56.422504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:56.422504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:56.422504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.422504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:56.422504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:56.422504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.422504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:56.422504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:56.422504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:56.422504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.422722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:56.422945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.422945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:56.423634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:56.423934 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.424248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:56.424248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.424248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:56.426179 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:56.426431 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:56.426557 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:56.426689 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:12:56.426981 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:56.427179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.427657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:56.427657 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.427657 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:56.427657 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:56.427657 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.427657 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:56.427657 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.427657 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:56.428437 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:56.428691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:56.428691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:56.428691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.428691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:56.428691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:56.428691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.428691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:56.428691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.428691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:56.429319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.429319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:56.429319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:56.429319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:56.429319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.429319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:56.429319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:56.429319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:12:56.429319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:56.429319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:56.429319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.429319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:56.429516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:56.429729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.429729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:56.429729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.429729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:56.430421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:56.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:56.430970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.430970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:56.430970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:56.430970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.430970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:56.430970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.430970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:56.433620 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:56.433856 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:56.433981 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:56.434129 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:56.434422 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:56.434649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.435197 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:56.435197 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.435197 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:56.435197 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:56.435197 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.435197 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:56.435197 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.435197 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:12:56.435955 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:12:56.436220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:56.436220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:12:56.436220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.436220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:12:56.436220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:12:56.436220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.436220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:12:56.436220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.436220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:12:56.436886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.436886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:56.436886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:56.436886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:56.436886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.436886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:56.436886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:56.436886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:56.436886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:56.436886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.436886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:56.437065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:56.437304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.437304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:56.437304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.437304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.438032 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:56.438330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:56.438599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.438599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:56.438599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.438599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:12:56.452640 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:12:56.453042 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:12:56.453200 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:12:56.453416 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:12:56.453748 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:56.453999 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:12:56.454221 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.454751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:12:56.454751 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.454751 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:12:56.454751 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:12:56.454751 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.454751 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:12:56.454751 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.454751 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:12:56.455609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.455609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:12:56.455609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:56.455609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:12:56.455609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.455609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:56.455609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:56.455609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.455609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:56.455609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:56.455609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:12:56.455609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.455822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:12:56.456029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.456029 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:12:56.456842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:12:56.457106 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:12:56.457419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:12:56.457419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:12:56.457419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:01.100602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:01.101102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:01.101102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:01.101780 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:01.131355 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:01.131595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.132073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:01.132073 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.132073 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:01.132073 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:01.132073 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.132073 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:01.132073 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.132073 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:01.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:01.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:01.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:01.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:01.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:01.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:01.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:01.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:01.132926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.133342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:01.133735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.133735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:01.133864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:01.134418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.135051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:01.135051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.135051 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:01.419940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:01.420890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:01.420890 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:01.421102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:01.421350 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:01.421415 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:01.421533 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:01.421630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:01.421732 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:01.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:01.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:01.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:01.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.422245 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:01.423533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.423533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:01.423533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:01.423533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:01.423533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.423533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:01.423533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:01.423533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.423533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:01.423533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:01.423533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:01.423533 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.423829 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:01.424062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.424062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:01.424421 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:01.425360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.426337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:01.426337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.426337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:01.426478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:01.426686 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:01.426762 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:01.426835 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:13:01.427155 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:01.427320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.427697 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:01.427697 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.427697 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:01.427697 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:01.427697 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.427697 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:01.427697 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.427697 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:01.428238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:01.428554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:01.428554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:01.428554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.428554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:01.428554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:01.428554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.428554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:01.428554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.428554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:01.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:01.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:01.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:01.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:01.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:01.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:13:01.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:01.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:01.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.428785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:01.429546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:01.429765 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:01.430074 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:01.430422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:01.430422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:01.430422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:01.430422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:01.430422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:01.430422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.430422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:01.430422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:01.430422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.430422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:01.430422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.430422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:01.433886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:01.434131 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:01.434275 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:01.434418 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:01.435074 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:01.436260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.436260 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:01.436260 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.436260 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:01.436260 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:01.436260 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.436260 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:01.436260 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.436260 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:01.436766 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:01.437217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:01.437217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:01.437217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.437217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:01.437217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:01.437217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.437217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:01.437217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.437217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:01.437399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.437399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:01.437399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:01.437399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:01.437399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.437399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:01.437399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:01.437399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:01.437399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:01.437399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.437399 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:01.437622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:01.437838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.437838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:01.437838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.437838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.438371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:01.438749 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:01.439134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.439134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:01.439134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.439134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:01.453034 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:01.453420 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:01.453420 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:01.453555 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:01.453918 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:01.454127 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:01.454357 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.454948 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:01.454948 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.454948 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:01.454948 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:01.454948 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.454948 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:01.454948 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.454948 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:01.455661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.455661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:01.455661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:01.455661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:01.455661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.455661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:01.455661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:01.455661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.455661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:01.455661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:01.455661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:01.455661 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.456210 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:01.456463 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:01.456722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:01.457085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:01.457085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:01.457085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:01.457085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:01.457085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:01.457085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:06.101048 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:06.101713 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:06.101713 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:06.102154 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:06.129456 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:06.129695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.130149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:06.130149 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.130149 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:06.130149 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:06.130149 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.130149 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:06.130149 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.130149 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:06.130956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.130956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:06.130956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:06.130956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:06.130956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.130956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:06.130956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:06.130956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.130956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:06.130956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:06.130956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:06.130956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.133306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:06.133699 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:06.134085 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.134730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:06.134730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:06.134730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.134730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:06.134730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.134730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:06.420563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:06.420873 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:06.420978 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:06.421833 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:06.422057 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:06.422216 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f10239e-f301-4769-9997-ba9637ac008d, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:06.422361 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:06.422473 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-33ca4fb3-7544-4ff4-b08a-81a0e7e69137, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:06.422589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.423062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:06.423062 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.423062 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:06.423062 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:06.423062 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.423062 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:06.423062 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.423062 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:06.423959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.423959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:06.423959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:06.423959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:06.423959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.423959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:06.423959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:06.423959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.423959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:06.423959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:06.423959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:06.423959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.424364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:06.424643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.424643 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:06.424729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:06.425134 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.425693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:06.425693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.425693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:06.426822 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:06.427079 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:06.427199 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:06.427328 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:13:06.427661 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:06.427819 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.428241 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:06.428241 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.428241 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:06.428241 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:06.428241 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.428241 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:06.428241 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.428241 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:06.428838 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:06.429561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:06.430000 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:06.430359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:06.430359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:06.430359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:06.430359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:06.430359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:06.430359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.430359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:06.430359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:06.430359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.430359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:06.430359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.430359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:06.430890 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:06.431602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:06.432320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.432320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:06.432320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:06.432320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.432320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:06.432320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.432320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:06.434632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:06.434867 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:06.434973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:06.435090 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:06.435416 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:06.435599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.436001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:06.436001 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.436001 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:06.436001 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:06.436001 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.436001 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:06.436001 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.436001 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:06.436636 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:06.437293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:06.437593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:06.437999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:06.437999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:06.437999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:06.437999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:06.437999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.437999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:06.437999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:06.437999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.437999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:06.437999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.437999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:06.437999 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:06.438315 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.438721 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:06.453491 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:06.453763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:06.453891 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:06.454066 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:06.454413 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:06.454693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:06.454854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.455381 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:06.455381 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.455381 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:06.455381 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:06.455381 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.455381 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:06.455381 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.455381 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:06.455936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.455936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:06.455936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:06.455936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:06.455936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.455936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:06.455936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:06.455936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.455936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:06.455936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:06.455936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:06.455936 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.456557 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:06.457272 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:06.457703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:06.458248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:06.458248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:06.458248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:06.458248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:06.458248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:06.458248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:11.101483 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:11.103820 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:11.103820 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:11.103820 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:11.132000 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:11.132345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:11.135166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:11.135662 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.136034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:11.136460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:11.137023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:11.137023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:11.137023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.137023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:11.137023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:11.137023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.137023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:11.137023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.137023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:11.425228 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:11.425228 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:11.425228 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:11.425467 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:11.425671 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:11.425814 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:11.425972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.427011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:11.427011 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.427011 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:11.427011 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:11.427011 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.427011 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:11.427011 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.427011 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:11.428014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.428014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:11.428014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:11.428014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:11.428014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.428014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:11.428014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:11.428014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.428014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:11.428014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:11.428014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:11.428014 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.428680 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:11.430693 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:11.431390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:11.432607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:11.432607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:11.432607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.432607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:11.432607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:11.432607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.432607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:11.432607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.432607 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:11.432607 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:11.432607 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:11.432607 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:13:11.432940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:11.432940 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.435330 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:11.435832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:11.436257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.436590 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:11.436929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.436929 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:11.438214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:11.442513 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:11.443277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.443277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:11.443277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:11.443277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.443277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:11.443277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.443277 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:11.443277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:11.443277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:11.443277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:11.443550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:11.443550 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:11.443550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.443550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:11.443550 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.443550 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:11.443550 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:11.443550 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.443550 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:11.443550 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.443550 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:11.443550 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:11.443846 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:11.444265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:11.444604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:11.444923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:11.444923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:11.444923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.444923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:11.444923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:11.444923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.444923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:11.444923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.444923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:11.444923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.444923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:11.444923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.944627 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.947736 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:11.951325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:11.951325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:11.951325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:11.951325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:11.951325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:11.951325 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.951488 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:11.951739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:11.951949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:11.952171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:11.953734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:11.953734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:11.953734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:11.953734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:11.953734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:11.953734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:11.953734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:11.953734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:11.953734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:11.953734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:11.953734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:11.953734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:16.102477 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:16.102828 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:16.102934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:16.103572 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:16.134601 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:16.134811 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.135530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:16.135530 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.135530 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:16.135530 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:16.135530 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.135530 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:16.135530 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.135530 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:16.137099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.137099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:16.137099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:16.137099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:16.137099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.137099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:16.137099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:16.137099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.137099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:16.137099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:16.137099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:16.137099 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.137324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:16.137555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.137555 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:16.138294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:16.138568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.138913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:16.138913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.138913 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:16.424640 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:16.424980 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:16.425096 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:16.425812 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:16.426001 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:16.426126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:16.426273 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.426698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:16.426698 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.426698 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:16.426698 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:16.426698 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.426698 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:16.426698 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.426698 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:16.427504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.427504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:16.427504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:16.427504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:16.427504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.427504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:16.427504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:16.427504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.427504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:16.427504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:16.427504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:16.427504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.427720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:16.427947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.427947 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:16.428168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:16.428443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.429022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:16.429022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.429022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:16.430343 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:16.430563 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:16.430644 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:16.430733 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:13:16.431095 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:16.431322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.431702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:16.431702 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.431702 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:16.431702 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:16.431702 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.431702 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:16.431702 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.431702 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:16.432245 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:16.432525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:16.432525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:16.432525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.432525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:16.432525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:16.432525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.432525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:16.432525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.432525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:16.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:16.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:16.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:16.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:16.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:16.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:13:16.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:16.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:16.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.432783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:16.433004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:16.433252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.433252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:16.433252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.433252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:16.433472 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:16.433769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:16.434053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.434053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:16.434053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:16.434053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.434053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:16.434053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.434053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:16.438619 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:16.438824 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:16.438926 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:16.439020 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:16.439330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:16.439498 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.439900 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:16.439900 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.439900 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:16.439900 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:16.439900 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.439900 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:16.439900 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.439900 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:16.440447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:16.440689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:16.440689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:16.440689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.440689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:16.440689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:16.440689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.440689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:16.440689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.440689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:16.440975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.440975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:16.440975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:16.440975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:16.440975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.440975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:16.440975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:16.440975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:16.440975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:16.440975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.440975 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:16.441205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:16.441442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.441442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:16.441442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.441442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.441665 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:16.441928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:16.442247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.442247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:16.442247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.442247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:16.454793 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:16.454985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:16.455065 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:16.455196 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:16.455478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:16.455657 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:16.455772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.456086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:16.456086 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.456086 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:16.456086 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:16.456086 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.456086 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:16.456086 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.456086 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:16.456673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.456673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:16.456673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:16.456673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:16.456673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.456673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:16.456673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:16.456673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.456673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:16.456673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:16.456673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:16.456673 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:16.457090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.457090 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:16.457294 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:16.457595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:16.457869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:16.457869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:16.457869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:21.102903 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:21.103303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:21.103303 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:21.103884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:21.124664 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=242 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:13:21.138373 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:21.138543 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:21.140136 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:21.140511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.140728 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:21.140961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:21.141295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:21.141295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:21.141295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.141295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:21.141295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:21.141295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.141295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:21.141295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.141295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:21.425274 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:21.425602 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:21.425686 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:21.426312 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:21.426514 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:21.426619 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:21.426702 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.427174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:21.427174 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.427174 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:21.427174 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:21.427174 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.427174 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:21.427174 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.427174 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:21.427965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.427965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:21.427965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:21.427965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:21.427965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.427965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:21.427965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:21.427965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.427965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:21.427965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:21.427965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:21.427965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.428242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:21.428434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.428434 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:21.428866 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:21.429201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.429529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:21.429529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.429529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:21.430594 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:21.430806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:21.430889 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:21.431154 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:13:21.431275 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:21.431456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.431826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:21.431826 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.431826 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:21.431826 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:21.431826 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.431826 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:21.431826 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.431826 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:21.432369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:21.432802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:21.432802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:21.432802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.432802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:21.432802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:21.432802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.432802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:21.432802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.432802 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:21.433070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.433070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:21.433070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:21.433070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:21.433070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.433070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:21.433070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:21.433070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:13:21.433070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:21.433070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:21.433070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.433070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:21.433456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:21.433860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:21.434291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:21.434821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:21.434821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:21.434821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:21.434821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:21.434821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:21.434821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.434821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:21.434821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:21.434821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.434821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:21.434821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.434821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:21.438942 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:21.439210 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:21.439338 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:21.439468 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:21.439806 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:21.440006 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.440501 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:21.440501 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.440501 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:21.440501 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:21.440501 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.440501 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:21.440501 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.440501 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:21.441198 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:21.441592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:21.441592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:21.441592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.441592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:21.441592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:21.441592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.441592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:21.441592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.441592 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:21.442055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.442055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:21.442055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:21.442055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:21.442055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.442055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:21.442055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:21.442055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:21.442055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:21.442055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.442055 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:21.442300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:21.442561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.442561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:21.442561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.442561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.442920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:21.443412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:21.443813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.443813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:21.443813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.443813 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:21.455245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:21.455504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:21.455632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:21.455786 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:21.456140 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:21.456357 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:21.456486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.456928 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:21.456928 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.456928 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:21.456928 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:21.456928 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.456928 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:21.456928 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.456928 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:21.457612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.457612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:21.457612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:21.457612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:21.457612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.457612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:21.457612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:21.457612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.457612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:21.457612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:21.457612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:21.457612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.457807 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:21.458073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.458073 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:21.458339 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:21.458593 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:21.458870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:21.458870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:21.458870 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:26.103243 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:26.103666 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:26.103761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:26.104366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:26.131390 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:26.131616 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.132266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:26.132266 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.132266 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:26.132266 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:26.132266 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.132266 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:26.132266 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.132266 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:26.132800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.132800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:26.132800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:26.132800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:26.132800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.132800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:26.132800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:26.132800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.132800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:26.132800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:26.132800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:26.132800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.133184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:26.133452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.133452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:26.133539 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:26.133831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.134177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:26.134177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.134177 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:26.425656 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:26.425946 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:26.425992 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:26.426506 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:26.426641 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:26.426892 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-30d72b29-5501-41ca-ae9f-1cdc7119aa42, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:26.426958 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.428754 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:26.428754 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.428754 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:26.428754 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:26.428754 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.428754 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:26.428754 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.428754 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:26.429430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.429430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:26.429430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:26.429430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:26.429430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.429430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:26.429430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:26.429430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.429430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:26.429430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:26.429430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:26.429430 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.429642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:26.429852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.429852 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:26.430092 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:26.430396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.430714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:26.430714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.430714 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:26.431893 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:26.431893 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:26.431893 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:26.431893 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:13:26.432839 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:26.432993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.439985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:26.439985 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.439985 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:26.439985 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:26.439985 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.439985 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:26.439985 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.439985 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:26.439985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:26.440898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:26.442566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:26.442949 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:26.443219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:26.443219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:26.443219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:26.443219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:26.443219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:26.443219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.443219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:26.443219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:26.443219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.443219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:26.443219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.443219 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:26.443442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:26.443711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:26.444034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.444034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:26.444034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:26.444034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.444034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:26.444034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.444034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:26.444034 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:26.444034 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:26.444034 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:26.444269 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:26.444269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.444269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:26.444269 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.444269 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:26.444269 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:26.444269 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.444269 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:26.444269 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.444269 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:26.444460 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:26.444691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:26.444691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:26.444691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.444691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:26.444691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:26.444691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.444691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:26.444691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.444691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:26.444982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.444982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:26.444982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:26.444982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:26.444982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.444982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:26.444982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:26.444982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:26.444982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:26.444982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.444982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:26.445225 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:26.445444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.445444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:26.445444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.445444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.445682 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:26.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:26.446358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.446358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:26.446358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.446358 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:26.455554 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:26.455781 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:26.455869 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:26.456007 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:26.456365 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:26.456606 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:26.456772 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.457384 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:26.457384 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.457384 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:26.457384 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:26.457384 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.457384 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:26.457384 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.457384 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:26.458053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.458053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:26.458053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:26.458053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:26.458053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.458053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:26.458053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:26.458053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.458053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:26.458053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:26.458053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:26.458053 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.458426 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:26.458676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.458676 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:26.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:26.460308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:26.460599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:26.460599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:26.460599 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:31.103738 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:31.104158 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:31.104158 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:31.104673 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:31.131571 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:31.131738 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.132153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:31.132153 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.132153 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:31.132153 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:31.132153 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.132153 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:31.132153 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.132153 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:31.132745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.132745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:31.132745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:31.132745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:31.132745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.132745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:31.132745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:31.132745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.132745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:31.132745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:31.132745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:31.132745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.132959 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:31.133174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.133174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:31.133496 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:31.133781 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.134077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:31.134077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.134077 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:31.426069 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:31.426722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:31.426837 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:31.427417 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:31.427580 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-d2699db0-e9de-4cd3-b18f-b5c822635662, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:31.427692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.428291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:31.428291 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.428291 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:31.428291 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:31.428291 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.428291 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:31.428291 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.428291 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:31.428965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.428965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:31.428965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:31.428965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:31.428965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.428965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:31.428965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:31.428965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.428965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:31.428965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:31.428965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:31.428965 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.429337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:31.429705 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:31.430132 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.430691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:31.430691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:31.430691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.430691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:31.430691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.430691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:31.431152 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:31.431412 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:31.431549 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:31.431709 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:13:31.432014 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:31.432353 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.432892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:31.432892 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.432892 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:31.432892 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:31.432892 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.432892 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:31.432892 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.432892 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:31.433645 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:31.434060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:31.434060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:31.434060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.434060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:31.434060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:31.434060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.434060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:31.434060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.434060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:31.434779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.434779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:31.434779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:31.434779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:31.434779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.434779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:31.434779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:31.434779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:13:31.434779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:31.434779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:31.434779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.434779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:31.435182 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:31.435578 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:31.435966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:31.436522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:31.436522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:31.436522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:31.436522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:31.436522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:31.436522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.436522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:31.436522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:31.436522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.436522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:31.436522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.436522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:31.442514 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:31.442807 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:31.442938 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:31.443079 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:31.443668 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:31.443875 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.446518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:31.447069 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:31.447537 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.447909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:31.448304 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:31.448739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:31.449278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:31.449278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.449278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:31.449278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:31.449278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.449278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:31.449278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.449278 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:31.455904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:31.456934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:31.456934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:31.456934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:31.456934 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:31.457093 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:31.457149 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:31.458254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:31.458660 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.458993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:31.458993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:31.458993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.458993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:31.458993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.458993 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:31.459368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:31.459769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:31.460142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:31.460142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:31.460142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:36.104091 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:36.104608 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:36.104608 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:36.105032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:36.131330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:36.131573 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.132203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:36.132203 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.132203 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:36.132203 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:36.132203 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.132203 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:36.132203 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.132203 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:36.132989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.132989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:36.132989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:36.132989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:36.132989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.132989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:36.132989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:36.132989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.132989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:36.132989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:36.132989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:36.132989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.133425 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:36.133794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.133794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:36.134059 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:36.134408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.134748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:36.134748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.134748 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:36.426722 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:36.427098 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:36.427204 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:36.427666 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:36.427808 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.428264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:36.428264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.428264 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:36.428264 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:36.428264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.428264 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:36.428264 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.428264 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:36.429271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.429271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:36.429271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:36.429271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:36.429271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.429271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:36.429271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:36.429271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.429271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:36.429271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:36.429271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:36.429271 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.429507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:36.429760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.429760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:36.430478 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:36.430800 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.431184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:36.431184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.431184 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:36.431481 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:36.431741 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:36.431866 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:36.432001 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:13:36.432389 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:36.432591 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.433050 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:36.433050 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.433050 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:36.433050 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:36.433050 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.433050 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:36.433050 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.433050 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:36.434170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:36.434520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:36.434520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:36.434520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.434520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:36.434520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:36.434520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.434520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:36.434520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.434520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:36.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:36.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:36.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:36.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:36.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:36.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:13:36.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:36.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:36.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:36.435214 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:36.435577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.435577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:36.435577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.435577 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:36.435833 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:36.436147 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:36.436454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.436454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:36.436454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:36.436454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.436454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:36.436454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.436454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:36.443009 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:36.443353 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:36.443489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:36.443663 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:36.443988 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:36.444168 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.444575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:36.444575 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.444575 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:36.444575 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:36.444575 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.444575 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:36.444575 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.444575 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:36.445081 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:36.445416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:36.445416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:36.445416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.445416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:36.445416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:36.445416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.445416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:36.445416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.445416 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:36.446596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.446596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:36.446596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:36.446596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:36.446596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.446596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:36.446596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:36.446596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:36.446596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:36.446596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.446596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:36.446810 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:36.447040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.447040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:36.447040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.447040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.447888 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:36.448288 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:36.448649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.448649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:36.448649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.448649 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:36.456275 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:36.456630 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:36.456747 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:36.456912 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:36.457253 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:36.457457 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:36.457568 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.457996 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:36.457996 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.457996 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:36.457996 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:36.457996 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.457996 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:36.457996 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.457996 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:36.459080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.459080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:36.459080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:36.459080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:36.459080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.459080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:36.459080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:36.459080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.459080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:36.459080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:36.459080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:36.459080 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.459390 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:36.459603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.459603 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:36.460333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:36.460700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:36.461007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:36.461007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:36.461007 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:41.104538 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:41.104998 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:41.104998 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:41.105574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:41.129757 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:41.129757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.130041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:41.130041 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.130041 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:41.130041 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:41.130041 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.130041 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:41.130041 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.130041 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:41.130941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.130941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:41.130941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:41.130941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:41.130941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.130941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:41.130941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:41.130941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.130941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:41.130941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:41.130941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:41.130941 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.131213 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:41.131458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.131458 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:41.132110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:41.132420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.132751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:41.132751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.132751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:41.427145 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:41.427573 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:41.427666 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:41.428250 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:41.428365 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:41.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:41.430459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.430826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:41.431253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:41.431778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:41.431778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:41.431778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:41.431778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.431778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:41.431778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:41.431778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.431778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:41.431778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.431778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:41.431778 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:41.432044 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:41.432044 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:41.432131 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:13:41.432450 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:41.432617 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.432991 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:41.432991 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.432991 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:41.432991 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:41.432991 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.432991 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:41.432991 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.432991 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:41.433549 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:41.433915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:41.433915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:41.433915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.433915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:41.433915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:41.433915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.433915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:41.433915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.433915 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:41.434137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.434137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:41.434137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:41.434137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:41.434137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.434137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:41.434137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:41.434137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:13:41.434137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:41.434137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:41.434137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.434137 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:41.434427 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:41.434735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.434735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:41.434735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.434735 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:41.434835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:41.435217 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:41.435626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.435626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:41.435626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:41.435626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.435626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:41.435626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.435626 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:41.443367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:41.443759 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:41.443862 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:41.443967 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:41.444302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:41.444511 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.444930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:41.444930 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.444930 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:41.444930 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:41.444930 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.444930 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:41.444930 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.444930 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:41.445544 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:41.445905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:41.445905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:41.445905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.445905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:41.445905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:41.445905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.445905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:41.445905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.445905 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:41.446139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.446139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:41.446139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:41.446139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:41.446139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.446139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:41.446139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:41.446139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:41.446139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:41.446139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.446139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:41.446412 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:41.446724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.446724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:41.446724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.446724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.446828 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:41.447232 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:41.447706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.447706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:41.447706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.447706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:41.456621 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:41.456884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:41.456973 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:41.457167 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:41.457489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:41.457693 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:41.457785 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.458275 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:41.458275 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.458275 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:41.458275 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:41.458275 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.458275 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:41.458275 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.458275 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:41.458889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.458889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:41.458889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:41.458889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:41.458889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.458889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:41.458889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:41.458889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.458889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:41.458889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:41.458889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:41.458889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.459215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:41.459520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.459520 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:41.459608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:41.459987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:41.460470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:41.460470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:41.460470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:46.104909 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:46.105445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:46.105445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:46.105961 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:46.131167 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:46.131347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.131791 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:46.131791 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.131791 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:46.131791 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:46.131791 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.131791 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:46.131791 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.131791 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:46.132470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.132470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:46.132470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:46.132470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:46.132470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.132470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:46.132470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:46.132470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.132470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:46.132470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:46.132470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:46.132470 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.132793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:46.133001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.133001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:46.133130 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:46.133440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.133769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:46.133769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.133769 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:46.427710 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:46.428030 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:46.428181 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:46.428767 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:46.428968 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.429452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:46.429452 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.429452 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:46.429452 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:46.429452 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.429452 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:46.429452 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.429452 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:46.430165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.430165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:46.430165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:46.430165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:46.430165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.430165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:46.430165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:46.430165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.430165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:46.430165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:46.430165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:46.430165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.430640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:46.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:46.431356 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.431692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:46.431692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:46.431692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.431692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:46.431692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.431692 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:46.432143 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:46.432404 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:46.432487 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:46.432569 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:13:46.432801 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:46.432944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.433308 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:46.433308 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.433308 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:46.433308 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:46.433308 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.433308 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:46.433308 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.433308 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:46.433767 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:46.434027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:46.434027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:46.434027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.434027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:46.434027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:46.434027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.434027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:46.434027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.434027 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:46.434397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.434397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:46.434397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:46.434397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:46.434397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.434397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:46.434397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:46.434397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:13:46.434397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:46.434397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:46.434397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.434397 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:46.434658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:46.434903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.434903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:46.434903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.434903 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:46.435066 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:46.435369 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:46.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:46.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:46.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:46.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:46.443827 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:46.444072 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:46.444199 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:46.444321 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:46.444570 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:46.444745 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.445086 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:46.445086 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.445086 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:46.445086 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:46.445086 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.445086 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:46.445086 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.445086 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:46.445600 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:46.445889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:46.445889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:46.445889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.445889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:46.445889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:46.445889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.445889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:46.445889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.445889 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:46.446325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.446325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:46.446325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:46.446325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:46.446325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.446325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:46.446325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:46.446325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:46.446325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:46.446325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.446325 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:46.446561 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:46.447087 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:46.447454 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:46.447789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:46.447789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:46.447789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:46.447789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:46.447789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.447789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:46.447789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:46.447789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.447789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:46.447789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.447789 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:46.456991 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:46.457277 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:46.457367 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:46.457686 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:46.457762 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:46.457942 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:46.458021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.458625 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:46.458625 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.458625 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:46.458625 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:46.458625 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.458625 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:46.458625 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.458625 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:46.459526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.459526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:46.459526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:46.459526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:46.459526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.459526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:46.459526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:46.459526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.459526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:46.459526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:46.459526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:46.459526 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.459760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:46.460023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.460023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:46.460346 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:46.460667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:46.460997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:46.460997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:46.460997 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:51.105338 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:51.105785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:51.105785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:51.106319 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:51.129913 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:51.130282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.130860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:51.130860 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.130860 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:51.130860 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:51.130860 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.130860 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:51.130860 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.130860 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:51.131821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.131821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:51.131821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:51.131821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:51.131821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.131821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:51.131821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:51.131821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.131821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:51.131821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:51.131821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:51.131821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.132026 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:51.132264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.132264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:51.132779 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:51.133360 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.133842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:51.133842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.133842 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:51.428109 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:51.428268 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:51.428363 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:51.428982 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:51.429205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.429684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:51.429684 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.429684 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:51.429684 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:51.429684 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.429684 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:51.429684 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.429684 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:51.430376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.430376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:51.430376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:51.430376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:51.430376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.430376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:51.430376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:51.430376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.430376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:51.430376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:51.430376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:51.430376 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.430575 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:51.430822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.430822 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:51.431065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:51.431760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.432335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:51.432335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.432335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:51.432433 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:51.432566 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:51.432638 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:51.432726 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:13:51.433062 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:51.433281 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.433698 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:51.433698 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.433698 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:51.433698 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:51.433698 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.433698 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:51.433698 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.433698 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:51.434201 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:51.434436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:51.434436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:51.434436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.434436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:51.434436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:51.434436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.434436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:51.434436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.434436 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:51.434740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.434740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:51.434740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:51.434740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:51.434740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.434740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:51.434740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:51.434740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:13:51.434740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:51.434740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:51.434740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.434740 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:51.434933 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:51.435142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.435142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:51.435142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.435142 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:51.435452 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:51.435722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:51.435984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.435984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:51.435984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:51.435984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.435984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:51.435984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.435984 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:51.444064 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:51.444229 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:51.444301 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:51.444410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:51.444763 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:51.445016 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.445473 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:51.445473 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.445473 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:51.445473 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:51.445473 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.445473 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:51.445473 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.445473 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:51.445989 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:51.446257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:51.446257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:51.446257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.446257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:51.446257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:51.446257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.446257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:51.446257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.446257 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:51.446594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.446594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:51.446594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:51.446594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:51.446594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.446594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:51.446594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:51.446594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:51.446594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:51.446594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.446594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:51.447001 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:51.447342 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:51.447556 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:51.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:51.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:51.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:51.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:51.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:51.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:51.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:51.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:51.457338 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:51.457523 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:51.457600 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:51.457745 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:51.458159 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:51.458391 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:51.458510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.458961 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:51.458961 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.458961 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:51.458961 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:51.458961 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.458961 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:51.458961 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.458961 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:51.459594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.459594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:51.459594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:51.459594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:51.459594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.459594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:51.459594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:51.459594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.459594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:51.459594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:51.459594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:51.459594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.459782 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:51.459973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.459973 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:51.460312 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:51.460558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:51.460827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:51.460827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:51.460827 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:56.106261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:56.106261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:56.106261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:56.106838 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:56.131970 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:56.131970 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.132171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:56.132171 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.132171 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:56.132171 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:56.132171 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.132171 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:56.132171 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.132171 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:56.133011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.133011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:56.133011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:56.133011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:56.133011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.133011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:56.133011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:56.133011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.133011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:56.133011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:56.133011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:56.133011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.133554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:56.133760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.133760 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:56.133945 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:56.134262 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.134566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:56.134566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.134566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:56.428484 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:56.428714 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:56.428805 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:56.429886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:56.429886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.430031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:56.430031 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.430031 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:56.430031 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:56.430031 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.430031 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:56.430031 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.430031 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:56.430776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.430776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:56.430776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:56.430776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:56.430776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.430776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:56.430776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:56.430776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.430776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:56.430776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:56.430776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:56.430776 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.430998 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:56.431244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.431244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:56.431524 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:56.432151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.432724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:56.432724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.432724 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:56.432724 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:56.432801 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:56.432925 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:56.432925 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:13:56.433312 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:56.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.434046 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:56.434046 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.434046 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:56.434046 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:56.434046 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.434046 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:56.434046 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.434046 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:56.434602 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:56.435267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:56.435587 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:56.435864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:56.435864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:56.435864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:56.435864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:56.435864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:56.435864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.435864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:56.435864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:56.435864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.435864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:56.435864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.435864 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:56.436131 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:56.436493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:56.437220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.437220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:56.437220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:56.437220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.437220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:56.437220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.437220 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:56.444331 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:56.444500 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:56.444574 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:56.444668 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:56.445585 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:56.445585 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.445719 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:56.445719 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.445719 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:56.445719 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:56.445719 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.445719 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:56.445719 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.445719 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:13:56.446279 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:56.446862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:56.447293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:56.447499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:56.447499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:56.447499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:56.447499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:56.447499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.447499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:56.447499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:56.447499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.447499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:56.447499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.447499 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.447894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:56.448254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:56.448531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.448531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:56.448531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.448531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:13:56.457592 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:13:56.457745 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:13:56.457947 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:13:56.458087 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:13:56.459261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:56.459261 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:13:56.459261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.459447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:13:56.459447 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.459447 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:13:56.459447 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:13:56.459447 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.459447 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:13:56.459447 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.459447 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:13:56.459930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.459930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:13:56.459930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:56.459930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:13:56.459930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.459930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:56.459930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:56.459930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.459930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:56.459930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:56.459930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:13:56.459930 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.460166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:13:56.460420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.460420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:13:56.460621 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:13:56.460899 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:13:56.461317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:13:56.461317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:13:56.461317 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:01.106137 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:01.106588 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:01.106588 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:01.107105 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:01.132099 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:01.132364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.132884 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:01.132884 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.132884 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:01.132884 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:01.132884 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.132884 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:01.132884 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.132884 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:01.133640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.133640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:01.133640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:01.133640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:01.133640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.133640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:01.133640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:01.133640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.133640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:01.133640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:01.133640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:01.133640 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.133835 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:01.134023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.134023 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:01.134522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:01.134795 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.135152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:01.135152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.135152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:01.428943 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:01.429141 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:01.429260 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:01.429884 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:01.430056 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.430707 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:01.430707 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.430707 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:01.430707 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:01.430707 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.430707 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:01.430707 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.430707 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:01.431447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.431447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:01.431447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:01.431447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:01.431447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.431447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:01.431447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:01.431447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.431447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:01.431447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:01.431447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:01.431447 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.431684 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:01.431894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.431894 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:01.432204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:01.432565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.432904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:01.432904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.432904 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:01.432904 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:01.432976 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:01.433060 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:01.433162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:14:01.433535 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:01.433704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.434110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:01.434110 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.434110 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:01.434110 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:01.434110 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.434110 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:01.434110 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.434110 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:01.434674 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:01.434925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:01.434925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:01.434925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.434925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:01.434925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:01.434925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.434925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:01.434925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.434925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:01.435266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.435266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:01.435266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:01.435266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:01.435266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.435266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:01.435266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:01.435266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:14:01.435266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:01.435266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:01.435266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.435266 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:01.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:01.435737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.435737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:01.435737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.435737 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:01.435943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:01.436248 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:01.436528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.436528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:01.436528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:01.436528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.436528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:01.436528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.436528 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:01.444646 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:01.444833 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:01.444914 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:01.445010 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:01.445522 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:01.445761 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.446231 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:01.446231 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.446231 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:01.446231 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:01.446231 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.446231 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:01.446231 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.446231 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:01.446824 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:01.447084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:01.447084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:01.447084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.447084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:01.447084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:01.447084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.447084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:01.447084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.447084 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:01.447457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.447457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:01.447457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:01.447457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:01.447457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.447457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:01.447457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:01.447457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:01.447457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:01.447457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.447457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:01.447642 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:01.447837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.447837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:01.447837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.447837 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.448139 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:01.448410 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:01.448703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.448703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:01.448703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.448703 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:01.457872 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:01.458032 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:01.458121 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:01.458272 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:01.458689 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:01.458916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:01.459009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.459525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:01.459525 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.459525 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:01.459525 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:01.459525 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.459525 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:01.459525 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.459525 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:01.460151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.460151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:01.460151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:01.460151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:01.460151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.460151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:01.460151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:01.460151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.460151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:01.460151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:01.460151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:01.460151 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.460364 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:01.460562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.460562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:01.460844 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:01.461170 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:01.461505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:01.461505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:01.461505 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:06.106534 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:06.107183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:06.107183 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:06.107657 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:06.130593 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:06.130809 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.131324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:06.131324 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.131324 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:06.131324 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:06.131324 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.131324 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:06.131324 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.131324 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:06.132207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.132207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:06.132207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:06.132207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:06.132207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.132207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:06.132207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:06.132207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.132207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:06.132207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:06.132207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:06.132207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.132622 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:06.132963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.132963 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:06.133350 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:06.133659 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.133966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:06.133966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.133966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:06.429359 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:06.429553 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:06.429643 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:06.430070 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:06.430261 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:06.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:06.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:06.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:06.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.430668 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:06.431299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.431299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:06.431299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:06.431299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:06.431299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.431299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:06.431299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:06.431299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.431299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:06.431299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:06.431299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:06.431299 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.431530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:06.431727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.431727 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:06.431966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:06.432586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.433165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:06.433165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.433165 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:06.433165 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:06.433271 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:06.433297 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:06.433400 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:14:06.433611 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:06.433744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:06.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:06.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:06.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:06.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.434037 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:06.434534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:06.434778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:06.434778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:06.434778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.434778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:06.434778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:06.434778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.434778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:06.434778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.434778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:06.435058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.435058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:06.435058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:06.435058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:06.435058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.435058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:06.435058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:06.435058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:14:06.435058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:06.435058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:06.435058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.435058 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:06.435534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:06.435972 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:06.436253 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:06.436529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:06.436529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:06.436529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:06.436529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:06.436529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:06.436529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.436529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:06.436529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:06.436529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.436529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:06.436529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.436529 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:06.444881 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:06.445022 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:06.445107 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:06.445232 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:06.445467 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:06.445633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.445925 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:06.445925 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.445925 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:06.445925 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:06.445925 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.445925 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:06.445925 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.445925 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:06.446422 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:06.446678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:06.446678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:06.446678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.446678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:06.446678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:06.446678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.446678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:06.446678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.446678 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:06.446966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.446966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:06.446966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:06.446966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:06.446966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.446966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:06.446966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:06.446966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:06.446966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:06.446966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.446966 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:06.447401 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:06.447831 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:06.448110 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:06.448433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:06.448433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:06.448433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:06.448433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:06.448433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.448433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:06.448433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:06.448433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.448433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:06.448433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.448433 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:06.458087 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:06.458259 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:06.458333 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:06.458464 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:06.458673 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:06.458819 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:06.458896 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.459264 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:06.459264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.459264 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:06.459264 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:06.459264 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.459264 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:06.459264 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.459264 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:06.459787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.459787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:06.459787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:06.459787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:06.459787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.459787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:06.459787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:06.459787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.459787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:06.459787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:06.459787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:06.459787 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.459979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:06.460181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.460181 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:06.460464 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:06.461010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:06.461420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:06.461420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:06.461420 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:11.106927 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:11.107672 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:11.107672 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:11.107954 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:11.132252 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:11.132252 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.132589 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:11.132589 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.132589 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:11.132589 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:11.132589 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.132589 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:11.132589 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.132589 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:11.133318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.133318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:11.133318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:11.133318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:11.133318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.133318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:11.133318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:11.133318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.133318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:11.133318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:11.133318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:11.133318 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.133612 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:11.133803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.133803 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:11.134022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:11.134757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.135320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:11.135320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.135320 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:11.429708 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:11.429942 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:11.430020 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:11.430587 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:11.430753 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.431202 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:11.431202 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.431202 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:11.431202 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:11.431202 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.431202 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:11.431202 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.431202 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:11.431926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.431926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:11.431926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:11.431926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:11.431926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.431926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:11.431926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:11.431926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.431926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:11.431926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:11.431926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:11.431926 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.432159 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:11.432368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.432368 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:11.432641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:11.433306 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.433633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:11.433633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.433633 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:11.433633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:11.433633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:11.433633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:11.433633 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:14:11.433896 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:11.433981 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.434348 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:11.434348 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.434348 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:11.434348 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:11.434348 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.434348 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:11.434348 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.434348 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:11.434862 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:11.435282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:11.435282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:11.435282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.435282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:11.435282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:11.435282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.435282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:11.435282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.435282 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:11.435440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.435440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:11.435440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:11.435440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:11.435440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.435440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:11.435440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:11.435440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:14:11.435440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:11.435440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:11.435440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.435440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:11.435630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:11.435836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.435836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:11.435836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.435836 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:11.436375 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:11.436956 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:11.437523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.437523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:11.437523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:11.437523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.437523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:11.437523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.437523 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:11.445158 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:11.445354 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:11.445429 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:11.445528 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:11.445815 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:11.445987 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.446395 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:11.446395 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.446395 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:11.446395 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:11.446395 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.446395 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:11.446395 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.446395 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:11.446942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:11.447230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:11.447230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:11.447230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.447230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:11.447230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:11.447230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.447230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:11.447230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.447230 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:11.447560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.447560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:11.447560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:11.447560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:11.447560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.447560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:11.447560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:11.447560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:11.447560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:11.447560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.447560 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:11.448011 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:11.448319 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:11.448547 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:11.448814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:11.448814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:11.448814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:11.448814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:11.448814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.448814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:11.448814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:11.448814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.448814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:11.448814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.448814 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:11.458327 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:11.458478 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:11.458551 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:11.458675 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:11.458925 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:11.459085 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:11.459178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.459565 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:11.459565 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.459565 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:11.459565 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:11.459565 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.459565 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:11.459565 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.459565 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:11.460156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.460156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:11.460156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:11.460156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:11.460156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.460156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:11.460156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:11.460156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.460156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:11.460156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:11.460156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:11.460156 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.460483 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:11.460666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.460666 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:11.460951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:11.461558 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:11.461923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:11.461923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:11.461923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:16.107371 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:16.109329 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:16.109329 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:16.109329 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:16.132306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:16.132494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.133256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:16.133256 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.133256 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:16.133256 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:16.133256 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.133256 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:16.133256 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.133256 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:16.133594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.133594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:16.133594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:16.133594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:16.133594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.133594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:16.133594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:16.133594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.133594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:16.133594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:16.133594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:16.133594 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.133921 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:16.134250 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:16.134700 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.135205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:16.135205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:16.135205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.135205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:16.135205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.135205 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:16.430162 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:16.430414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:16.430504 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:16.430892 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:16.431031 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.431516 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:16.431516 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.431516 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:16.431516 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:16.431516 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.431516 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:16.431516 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.431516 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:16.432070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.432070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:16.432070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:16.432070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:16.432070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.432070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:16.432070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:16.432070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.432070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:16.432070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:16.432070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:16.432070 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.432527 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:16.432722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.432722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:16.432909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:16.433554 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.433830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:16.433830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.433830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:16.433830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:16.433830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:16.433830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:16.433830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:14:16.434258 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:16.434258 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.434480 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:16.434480 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.434480 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:16.434480 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:16.434480 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.434480 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:16.434480 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.434480 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:16.434942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:16.435293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:16.435293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:16.435293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.435293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:16.435293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:16.435293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.435293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:16.435293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.435293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:16.435634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.435634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:16.435634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:16.435634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:16.435634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.435634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:16.435634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:16.435634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:14:16.435634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:16.435634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:16.435634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.435634 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:16.436223 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:16.436441 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:16.436672 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:16.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:16.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:16.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:16.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:16.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:16.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:16.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:16.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:16.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.436951 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:16.445434 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:16.445591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:16.445715 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:16.445779 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:16.446021 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:16.446229 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.446567 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:16.446567 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.446567 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:16.446567 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:16.446567 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.446567 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:16.446567 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.446567 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:16.447062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:16.447344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:16.447344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:16.447344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.447344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:16.447344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:16.447344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.447344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:16.447344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.447344 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:16.447689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.447689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:16.447689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:16.447689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:16.447689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.447689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:16.447689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:16.447689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:16.447689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:16.447689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.447689 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:16.447886 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:16.448111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.448111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:16.448111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.448111 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.448456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:16.448730 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:16.449022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.449022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:16.449022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.449022 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:16.458541 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:16.458679 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:16.458773 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:16.458888 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:16.459103 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:16.459292 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:16.459374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.459695 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:16.459695 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.459695 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:16.459695 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:16.459695 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.459695 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:16.459695 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.459695 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:16.460301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.460301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:16.460301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:16.460301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:16.460301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.460301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:16.460301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:16.460301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.460301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:16.460301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:16.460301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:16.460301 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:16.460725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.460725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:16.461044 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:16.461331 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:16.461608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:16.461608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:16.461608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:21.108299 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:21.108871 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:21.108871 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:21.109525 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:21.130051 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG dbcounter [-] [89255] Writing DB stats neutron:SELECT=246 {{(pid=89255) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 09 01:14:21.134247 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:21.134247 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.134783 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:21.134783 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.134783 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:21.134783 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:21.134783 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.134783 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:21.134783 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.134783 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:21.135493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.135493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:21.135493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:21.135493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:21.135493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.135493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:21.135493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:21.135493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.135493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:21.135493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:21.135493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:21.135493 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.135793 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:21.135995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.135995 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:21.136254 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:21.136510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:21.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.136778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:21.430566 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:21.430732 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:21.430828 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:21.431459 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:21.431667 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.432206 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:21.432206 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.432206 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:21.432206 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:21.432206 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.432206 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:21.432206 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.432206 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:21.432826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.432826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:21.432826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:21.432826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:21.432826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.432826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:21.432826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:21.432826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.432826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:21.432826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:21.432826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:21.432826 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.433048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:21.433291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.433291 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:21.433566 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:21.434150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.434701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:21.434701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.434701 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:21.434701 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:21.434701 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:21.434701 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:21.434701 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:14:21.434701 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:21.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:21.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:21.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:21.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:21.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.434834 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:21.435305 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:21.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:21.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:21.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:21.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:21.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:21.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.435569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:21.435845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.435845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:21.435845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:21.435845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:21.435845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.435845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:21.435845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:21.435845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:14:21.435845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:21.435845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:21.435845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.435845 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:21.436293 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:21.436757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:21.437012 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:21.437322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:21.437322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:21.437322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:21.437322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:21.437322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:21.437322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.437322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:21.437322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:21.437322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.437322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:21.437322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.437322 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:21.445661 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:21.445820 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:21.445898 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:21.445985 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:21.446252 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:21.446450 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.446832 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:21.446832 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.446832 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:21.446832 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:21.446832 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.446832 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:21.446832 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.446832 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:21.447345 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:21.447569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:21.447569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:21.447569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.447569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:21.447569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:21.447569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.447569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:21.447569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.447569 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:21.447908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.447908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:21.447908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:21.447908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:21.447908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.447908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:21.447908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:21.447908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:21.447908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:21.447908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.447908 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:21.448105 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:21.448337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.448337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:21.448337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.448337 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.448658 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:21.449407 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:21.449898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.449898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:21.449898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.449898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:21.458818 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:21.459055 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:21.459203 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:21.459383 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:21.459632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:21.459832 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:21.459944 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.460402 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:21.460402 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.460402 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:21.460402 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:21.460402 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.460402 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:21.460402 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.460402 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:21.460939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.460939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:21.460939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:21.460939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:21.460939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.460939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:21.460939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:21.460939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.460939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:21.460939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:21.460939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:21.460939 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.461161 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:21.461371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.461371 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:21.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:21.462176 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:21.462522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:21.462522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:21.462522 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:26.108710 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:26.109339 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:26.109339 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:26.109591 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:26.133490 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:26.133651 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.134204 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:26.134204 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.134204 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:26.134204 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:26.134204 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.134204 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:26.134204 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.134204 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:26.134865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.134865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:26.134865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:26.134865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:26.134865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.134865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:26.134865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:26.134865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.134865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:26.134865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:26.134865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:26.134865 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.135097 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:26.135459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.135459 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:26.135563 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:26.136153 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.136675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:26.136675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.136675 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:26.430980 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:26.431173 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:26.431245 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:26.431683 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:26.431854 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.432336 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:26.432336 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.432336 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:26.432336 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:26.432336 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.432336 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:26.432336 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.432336 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:26.432920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.432920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:26.432920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:26.432920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:26.432920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.432920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:26.432920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:26.432920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.432920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:26.432920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:26.432920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:26.432920 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.433295 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:26.433671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.433671 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:26.433755 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:26.434292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.434764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:26.434764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.434764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:26.434764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:26.434764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:26.434764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:26.434764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:14:26.434764 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:26.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:26.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:26.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:26.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:26.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.434979 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:26.435525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:26.436065 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:26.436618 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:26.437042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:26.437042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:26.437042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:26.437042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:26.437042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:26.437042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.437042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:26.437042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:26.437042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.437042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:26.437042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.437042 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:26.437530 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:26.437821 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:26.438146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.438146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:26.438146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:26.438146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.438146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:26.438146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.438146 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:26.445881 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:26.446023 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:26.446112 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:26.446208 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:26.446473 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:26.446716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.447152 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:26.447152 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.447152 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:26.447152 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:26.447152 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.447152 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:26.447152 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.447152 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:26.447657 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:26.447892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:26.447892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:26.447892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.447892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:26.447892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:26.447892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.447892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:26.447892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.447892 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:26.448200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.448200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:26.448200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:26.448200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:26.448200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.448200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:26.448200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:26.448200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:26.448200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:26.448200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.448200 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:26.448396 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:26.448595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.448595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:26.448595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.448595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.448917 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:26.449203 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:26.449476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.449476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:26.449476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.449476 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:26.459115 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:26.459306 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:26.459414 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:26.459547 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:26.459827 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:26.460064 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:26.460199 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.460586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:26.460586 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.460586 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:26.460586 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:26.460586 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.460586 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:26.460586 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.460586 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:26.461171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.461171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:26.461171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:26.461171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:26.461171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.461171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:26.461171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:26.461171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.461171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:26.461171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:26.461171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:26.461171 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.461418 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:26.461608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.461608 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:26.461857 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:26.462124 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:26.462681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:26.462681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:26.462681 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:31.109119 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:31.109661 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:31.109661 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:31.109933 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:31.137267 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:31.137267 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.137670 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:31.137670 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.137670 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:31.137670 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:31.137670 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.137670 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:31.137670 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.137670 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:31.138388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.138388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:31.138388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:31.138388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:31.138388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.138388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:31.138388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:31.138388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.138388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:31.138388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:31.138388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:31.138388 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.138628 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:31.138840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.138840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:31.139256 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:31.139898 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.140222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:31.140222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.140222 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:31.431374 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:31.431834 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:31.431965 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:31.432632 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:31.432860 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.433479 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:31.433479 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.433479 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:31.433479 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:31.433479 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.433479 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:31.433479 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.433479 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:31.434586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.434586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:31.434586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:31.434586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:31.434586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.434586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:31.434586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:31.434586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.434586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:31.434586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:31.434586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:31.434586 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:31.435239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.435239 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:31.435239 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:31.435971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:31.436604 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.437359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:31.437359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.437359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:31.437359 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:31.437359 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:31.437359 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:14:31.437359 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:31.437359 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.437531 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:31.437531 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.437531 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:31.437531 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:31.437531 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.437531 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:31.437531 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.437531 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:31.438265 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:31.438630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:31.438630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:31.438630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.438630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:31.438630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:31.438630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.438630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:31.438630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.438630 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:31.439207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.439207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:31.439207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:31.439207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:31.439207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.439207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:31.439207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:31.439207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:14:31.439207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:31.439207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:31.439207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.439207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:31.439486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:31.439751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.439751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:31.439751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.439751 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:31.440324 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:31.440638 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:31.440960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.440960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:31.440960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:31.440960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.440960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:31.440960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.440960 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:31.446202 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:31.446435 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:31.446568 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:31.446720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:31.447130 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:31.447354 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.447840 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:31.447840 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.447840 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:31.447840 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:31.447840 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.447840 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:31.447840 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.447840 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:31.448744 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:31.449270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:31.449270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:31.449270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.449270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:31.449270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:31.449270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.449270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:31.449270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.449270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:31.449718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.449718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:31.449718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:31.449718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:31.449718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.449718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:31.449718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:31.449718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:31.449718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:31.449718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.449718 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:31.450118 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:31.450546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:31.450794 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:31.451115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:31.451115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:31.451115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:31.451115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:31.451115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.451115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:31.451115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:31.451115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.451115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:31.451115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.451115 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:31.459410 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:31.459618 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:31.459704 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:31.459856 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:31.460286 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:31.460438 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:31.460534 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.460942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:31.460942 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.460942 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:31.460942 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:31.460942 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.460942 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:31.460942 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.460942 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:31.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:31.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:31.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:31.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:31.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:31.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:31.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:31.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:31.461595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.461830 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:31.462041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.462041 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:31.462333 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:31.462710 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:31.463045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:31.463045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:31.463045 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:36.109475 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:36.109924 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:36.109924 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:36.110479 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:36.134290 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:36.134510 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.135009 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:36.135009 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.135009 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:36.135009 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:36.135009 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.135009 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:36.135009 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.135009 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:36.135706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.135706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:36.135706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:36.135706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:36.135706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.135706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:36.135706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:36.135706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.135706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:36.135706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:36.135706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:36.135706 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.135943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:36.136300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.136300 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:36.136442 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:36.137021 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.137456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:36.137456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.137456 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:36.431853 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:36.432278 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:36.432473 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:36.432869 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:36.433040 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.433514 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:36.433514 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.433514 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:36.433514 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:36.433514 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.433514 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:36.433514 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.433514 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:36.434208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.434208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:36.434208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:36.434208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:36.434208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.434208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:36.434208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:36.434208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.434208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:36.434208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:36.434208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:36.434208 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.434545 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:36.434847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.434847 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:36.434909 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:36.436166 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.436720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:36.436720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.436720 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:36.436720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:36.436720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:36.436720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:36.436720 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:14:36.436912 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:36.437340 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.437504 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:36.437504 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.437504 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:36.437504 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:36.437504 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.437504 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:36.437504 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.437504 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:36.438004 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:36.438609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:36.438943 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:36.439269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:36.439269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:36.439269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:36.439269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:36.439269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:36.439269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.439269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:36.439269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:36.439269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.439269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:36.439269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.439269 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:36.439559 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:36.440307 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:36.441215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.441215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:36.441215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:36.441215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.441215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:36.441215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.441215 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:36.446566 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:36.446833 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:36.446916 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:36.447059 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:36.447434 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:36.447624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.448062 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:36.448062 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.448062 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:36.448062 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:36.448062 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.448062 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:36.448062 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.448062 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:36.448654 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:36.448992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:36.448992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:36.448992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.448992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:36.448992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:36.448992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.448992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:36.448992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.448992 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:36.449238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.449238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:36.449238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:36.449238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:36.449238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.449238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:36.449238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:36.449238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:36.449238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:36.449238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.449238 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:36.449503 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:36.449818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.449818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:36.449818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.449818 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.449923 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:36.450347 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:36.450764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.450764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:36.450764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.450764 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:36.459788 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:36.460084 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:36.460165 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:36.460337 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:36.460681 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:36.460886 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:36.460976 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.461408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:36.461408 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.461408 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:36.461408 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:36.461408 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.461408 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:36.461408 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.461408 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:36.462010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.462010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:36.462010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:36.462010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:36.462010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.462010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:36.462010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:36.462010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.462010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:36.462010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:36.462010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:36.462010 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.462335 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:36.462639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.462639 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:36.462704 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:36.463049 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:36.463486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:36.463486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:36.463486 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:41.109913 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:41.110402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:41.110402 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:41.111058 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:41.135761 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:41.135971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.136518 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:41.136518 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.136518 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:41.136518 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:41.136518 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.136518 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:41.136518 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.136518 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:41.138135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.138135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:41.138135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:41.138135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:41.138135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.138135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:41.138135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:41.138135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.138135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:41.138135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:41.138135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:41.138135 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.138497 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:41.138778 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:41.139071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.139387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:41.139387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:41.139387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.139387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:41.139387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.139387 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:41.432366 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:41.433241 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:41.433241 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:41.433439 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:41.433609 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:41.435242 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:41.435595 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.435806 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:41.436060 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:41.436377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:41.436377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:41.436377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:41.436377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.436377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:41.436377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:41.436377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.436377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:41.436377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.436377 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:41.436544 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:41.436733 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:41.436733 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:41.436842 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:14:41.437489 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:41.437489 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.437624 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:41.437624 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.437624 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:41.437624 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:41.437624 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.437624 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:41.437624 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.437624 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:41.438098 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:41.438653 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:41.439178 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:41.439641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:41.439641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:41.439641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:41.439641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:41.439641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:41.439641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.439641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:41.439641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:41.439641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.439641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:41.439641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.439641 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:41.440048 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:41.440546 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:41.440804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.440804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:41.440804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:41.440804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.440804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:41.440804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.440804 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:41.446839 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:41.447017 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:41.447102 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:41.447207 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:41.447856 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:41.447856 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.447979 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:41.447979 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.447979 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:41.447979 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:41.447979 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.447979 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:41.447979 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.447979 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:41.448494 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:41.448734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:41.448734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:41.448734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.448734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:41.448734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:41.448734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.448734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:41.448734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.448734 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:41.449224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.449224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:41.449224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:41.449224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:41.449224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.449224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:41.449224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:41.449224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:41.449224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:41.449224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.449224 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:41.449637 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:41.449942 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:41.450158 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:41.450448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:41.450448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:41.450448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:41.450448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:41.450448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.450448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:41.450448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:41.450448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.450448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:41.450448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.450448 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:41.460081 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:41.460302 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:41.460399 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:41.460532 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:41.461244 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:41.461244 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:41.461244 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.461440 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:41.461440 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.461440 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:41.461440 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:41.461440 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.461440 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:41.461440 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.461440 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:41.462034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.462034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:41.462034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:41.462034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:41.462034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.462034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:41.462034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:41.462034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.462034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:41.462034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:41.462034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:41.462034 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.462408 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:41.462729 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:41.463126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:41.463598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:41.463598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:41.463598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:41.463598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:41.463598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:41.463598 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:46.110292 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:46.110779 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:46.110779 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:46.111445 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:46.135830 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:46.136025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.136562 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:46.136562 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.136562 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:46.136562 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:46.136562 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.136562 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:46.136562 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.136562 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:46.137292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.137292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:46.137292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:46.137292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:46.137292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.137292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:46.137292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:46.137292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.137292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:46.137292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:46.137292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:46.137292 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.137525 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:46.137711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.137711 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:46.138025 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:46.138691 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.139207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:46.139207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.139207 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:46.432899 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:46.433065 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:46.433175 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:46.433594 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-bc72357e-b779-467d-a886-8cb010ab0d53, routes=[], if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:46.433725 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.434126 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:46.434126 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.434126 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:46.434126 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:46.434126 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.434126 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:46.434126 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.434126 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:46.434739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.434739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:46.434739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:46.434739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:46.434739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.434739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:46.434739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:46.434739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.434739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:46.434739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:46.434739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:46.434739 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.434988 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:46.435185 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.435185 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:46.435507 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:46.436064 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.436596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:46.436596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.436596 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:46.436726 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:46.436856 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:46.436940 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:46.437013 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89255) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1163}} Jun 09 01:14:46.437254 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:46.437385 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.437688 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:46.437688 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.437688 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:46.437688 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:46.437688 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.437688 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:46.437688 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.437688 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:46.438150 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:46.438443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:46.438443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:46.438443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.438443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:46.438443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:46.438443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.438443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:46.438443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.438443 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:46.438757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.438757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:46.438757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:46.438757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:46.438757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.438757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:46.438757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:46.438757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 09 01:14:46.438757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:46.438757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:46.438757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.438757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:46.439180 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:46.439500 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1167, in set_fip_distributed_flag Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^ Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:46.439713 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:46.439982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:46.439982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:46.439982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:46.439982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:46.439982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:46.439982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.439982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:46.439982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:46.439982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.439982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:46.439982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.439982 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:46.447126 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:46.447330 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:46.447423 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:46.447531 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:46.447785 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:46.447971 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.448419 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:46.448419 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.448419 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:46.448419 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:46.448419 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.448419 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:46.448419 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.448419 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 09 01:14:46.449174 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 09 01:14:46.449444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:46.449444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 09 01:14:46.449444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.449444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 09 01:14:46.449444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 09 01:14:46.449444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.449444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 09 01:14:46.449444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.449444 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.command Jun 09 01:14:46.450071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.450071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:46.450071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:46.450071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:46.450071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.450071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:46.450071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:46.450071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:46.450071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:46.450071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.450071 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:46.450374 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:46.450571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.450571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:46.450571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.450571 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.451179 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1228, in update_ha_failover Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:46.451457 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:46.451722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.451722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:46.451722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.451722 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Jun 09 01:14:46.460320 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89255) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 09 01:14:46.460522 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 09 01:14:46.460641 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89255) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 09 01:14:46.460802 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89255) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:963}} Jun 09 01:14:46.461043 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:46.461269 np0000002381 neutron-ovn-maintenance-worker[89255]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89255) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 09 01:14:46.461351 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.461716 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Traceback (most recent call last): Jun 09 01:14:46.461716 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.461716 np0000002381 neutron-ovn-maintenance-worker[89255]: txn.results.put(txn.do_commit()) Jun 09 01:14:46.461716 np0000002381 neutron-ovn-maintenance-worker[89255]: ^^^^^^^^^^^^^^^ Jun 09 01:14:46.461716 np0000002381 neutron-ovn-maintenance-worker[89255]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.461716 np0000002381 neutron-ovn-maintenance-worker[89255]: raise RuntimeError(msg) Jun 09 01:14:46.461716 np0000002381 neutron-ovn-maintenance-worker[89255]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.461716 np0000002381 neutron-ovn-maintenance-worker[89255]: Jun 09 01:14:46.462551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.462551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 09 01:14:46.462551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:46.462551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 09 01:14:46.462551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.462551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:46.462551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:46.462551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.462551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:46.462551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:46.462551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 09 01:14:46.462551 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.462757 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 09 01:14:46.462937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.462937 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics [None req-31605486-dc84-418a-8119-5d0de3b45378 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics Traceback (most recent call last): Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics work() Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(*args, **kwargs) Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 09 01:14:46.463548 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 273, in transaction Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics next(self.gen) Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics with self.create_transaction( Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics self.result = self.commit() Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise result.ex Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 09 01:14:46.463869 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 09 01:14:46.464270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics raise RuntimeError(msg) Jun 09 01:14:46.464270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 09 01:14:46.464270 np0000002381 neutron-ovn-maintenance-worker[89255]: ERROR futurist.periodics